2023-10-18T15:54:05.316 INFO:root:teuthology version: 0.0.1.dev197+g8cdab07 2023-10-18T15:54:05.317 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --name vshankar-2023-10-16_12:48:28-fs-wip-vshankar-testing-pacific-20231016.091648-testing-default-smithi --archive /home/teuthworker/archive/vshankar-2023-10-16_12:48:28-fs-wip-vshankar-testing-pacific-20231016.091648-testing-default-smithi/7429067 --description fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} --owner scheduled_vshankar@teuthology -- /home/teuthworker/archive/vshankar-2023-10-16_12:48:28-fs-wip-vshankar-testing-pacific-20231016.091648-testing-default-smithi/7429067/orig.config.yaml 2023-10-18T15:54:05.361 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-18T15:54:05.452 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/vshankar-2023-10-16_12:48:28-fs-wip-vshankar-testing-pacific-20231016.091648-testing-default-smithi/7429067 branch: wip-vshankar-testing-pacific-20231016.091648 description: fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} email: vshankar@redhat.com first_in_suite: false job_id: '7429067' kernel: client: sha1: distro flavor: default kdb: true sha1: 47672119149b103ec7cec2cc65c063ccbb001ac7 last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: vshankar-2023-10-16_12:48:28-fs-wip-vshankar-testing-pacific-20231016.091648-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.4' overrides: admin_socket: branch: wip-vshankar-testing-pacific-20231016.091648 ceph: cephfs: max_mds: 3 session_timeout: 300 conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds oft prefetch dirfrags: false mds op complaint time: 180 mds verify scatter: true mds_sleep_rank_change: 5000000.0 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - Replacing daemon mds sha1: b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 check-counter: counters: mds: - mds.exported - mds.imported install: ceph: flavor: default sha1: b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-vshankar-testing-pacific-20231016.091648 sha1: b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5 owner: scheduled_vshankar@teuthology priority: 80 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - mds.e - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - mds.d - mds.f - osd.4 - osd.5 - osd.6 - osd.7 seed: 7825 sha1: b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5 sleep_before_teardown: 0 subset: 33/1024 suite: fs suite_branch: wip-vshankar-testing-pacific-20231016.091648 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5 targets: smithi088.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDM0J7RzHKtMvi+MMC04dICQhoKb8N07LvmP9S2NyG0Ljq9D1SSqj3y5PRJLxhMdmsMjbaiAzPfrTsQdrK9x9Ho= smithi151.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBbNF+1REZY6QhkJY1gSEnsFbRSKC+civjX7dXcQZ0VlbSAYkufd5XtNVsJX3hiJyfcxEIIh87QC/wsq4npNqzM= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - ceph: null - kclient: null - mds_thrash: null - workunit: clients: all: - fs/snaps teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 8cdab074dcca9a68965bc5a50e9c30b691949723 timestamp: 2023-10-16_12:48:28 tube: smithi user: vshankar verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-10-18T15:54:05.452 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa; will attempt to use it 2023-10-18T15:54:05.453 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks 2023-10-18T15:54:05.453 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-18T15:54:05.455 INFO:teuthology.task.internal:Checking packages... 2023-10-18T15:54:05.478 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash 'b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5' 2023-10-18T15:54:05.478 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-18T15:54:05.479 INFO:teuthology.packaging:ref: None 2023-10-18T15:54:05.479 INFO:teuthology.packaging:tag: None 2023-10-18T15:54:05.479 INFO:teuthology.packaging:branch: wip-vshankar-testing-pacific-20231016.091648 2023-10-18T15:54:05.479 INFO:teuthology.packaging:sha1: b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5 2023-10-18T15:54:05.480 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-vshankar-testing-pacific-20231016.091648 2023-10-18T15:54:05.647 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-100.gb4ddd52c 2023-10-18T15:54:05.647 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-18T15:54:05.654 INFO:teuthology.task.internal:no buildpackages task found 2023-10-18T15:54:05.654 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-18T15:54:05.691 INFO:teuthology.task.internal:Saving configuration 2023-10-18T15:54:05.707 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-18T15:54:05.716 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-18T15:54:05.743 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi088.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2023-10-16_12:48:28-fs-wip-vshankar-testing-pacific-20231016.091648-testing-default-smithi/7429067', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-18 15:48:36.620568', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDM0J7RzHKtMvi+MMC04dICQhoKb8N07LvmP9S2NyG0Ljq9D1SSqj3y5PRJLxhMdmsMjbaiAzPfrTsQdrK9x9Ho='} 2023-10-18T15:54:05.765 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi151.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2023-10-16_12:48:28-fs-wip-vshankar-testing-pacific-20231016.091648-testing-default-smithi/7429067', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-18 15:48:36.618906', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBbNF+1REZY6QhkJY1gSEnsFbRSKC+civjX7dXcQZ0VlbSAYkufd5XtNVsJX3hiJyfcxEIIh87QC/wsq4npNqzM='} 2023-10-18T15:54:05.765 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-18T15:54:05.773 INFO:teuthology.task.internal:roles: ubuntu@smithi088.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2023-10-18T15:54:05.774 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-10-18T15:54:05.774 INFO:teuthology.run_tasks:Running task console_log... 2023-10-18T15:54:05.871 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fb56bec4a60>, signals=[15]) 2023-10-18T15:54:05.873 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-18T15:54:05.880 INFO:teuthology.task.internal:Opening connections... 2023-10-18T15:54:05.881 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi088.front.sepia.ceph.com 2023-10-18T15:54:05.882 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T15:54:05.951 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi151.front.sepia.ceph.com 2023-10-18T15:54:05.952 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T15:54:06.021 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-18T15:54:06.028 DEBUG:teuthology.orchestra.run.smithi088:> uname -m 2023-10-18T15:54:06.044 INFO:teuthology.orchestra.run.smithi088.stdout:x86_64 2023-10-18T15:54:06.045 DEBUG:teuthology.orchestra.run.smithi088:> cat /etc/os-release 2023-10-18T15:54:06.100 INFO:teuthology.orchestra.run.smithi088.stdout:NAME="Red Hat Enterprise Linux" 2023-10-18T15:54:06.100 INFO:teuthology.orchestra.run.smithi088.stdout:VERSION="8.4 (Ootpa)" 2023-10-18T15:54:06.101 INFO:teuthology.orchestra.run.smithi088.stdout:ID="rhel" 2023-10-18T15:54:06.101 INFO:teuthology.orchestra.run.smithi088.stdout:ID_LIKE="fedora" 2023-10-18T15:54:06.101 INFO:teuthology.orchestra.run.smithi088.stdout:VERSION_ID="8.4" 2023-10-18T15:54:06.101 INFO:teuthology.orchestra.run.smithi088.stdout:PLATFORM_ID="platform:el8" 2023-10-18T15:54:06.101 INFO:teuthology.orchestra.run.smithi088.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2023-10-18T15:54:06.102 INFO:teuthology.orchestra.run.smithi088.stdout:ANSI_COLOR="0;31" 2023-10-18T15:54:06.102 INFO:teuthology.orchestra.run.smithi088.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2023-10-18T15:54:06.102 INFO:teuthology.orchestra.run.smithi088.stdout:HOME_URL="https://www.redhat.com/" 2023-10-18T15:54:06.102 INFO:teuthology.orchestra.run.smithi088.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-10-18T15:54:06.102 INFO:teuthology.orchestra.run.smithi088.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-18T15:54:06.103 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T15:54:06.103 INFO:teuthology.orchestra.run.smithi088.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-18T15:54:06.103 INFO:teuthology.orchestra.run.smithi088.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2023-10-18T15:54:06.103 INFO:teuthology.orchestra.run.smithi088.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-10-18T15:54:06.103 INFO:teuthology.orchestra.run.smithi088.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2023-10-18T15:54:06.104 INFO:teuthology.lock.ops:Updating smithi088.front.sepia.ceph.com on lock server 2023-10-18T15:54:06.127 DEBUG:teuthology.orchestra.run.smithi151:> uname -m 2023-10-18T15:54:06.144 INFO:teuthology.orchestra.run.smithi151.stdout:x86_64 2023-10-18T15:54:06.144 DEBUG:teuthology.orchestra.run.smithi151:> cat /etc/os-release 2023-10-18T15:54:06.200 INFO:teuthology.orchestra.run.smithi151.stdout:NAME="Red Hat Enterprise Linux" 2023-10-18T15:54:06.201 INFO:teuthology.orchestra.run.smithi151.stdout:VERSION="8.4 (Ootpa)" 2023-10-18T15:54:06.201 INFO:teuthology.orchestra.run.smithi151.stdout:ID="rhel" 2023-10-18T15:54:06.201 INFO:teuthology.orchestra.run.smithi151.stdout:ID_LIKE="fedora" 2023-10-18T15:54:06.201 INFO:teuthology.orchestra.run.smithi151.stdout:VERSION_ID="8.4" 2023-10-18T15:54:06.202 INFO:teuthology.orchestra.run.smithi151.stdout:PLATFORM_ID="platform:el8" 2023-10-18T15:54:06.202 INFO:teuthology.orchestra.run.smithi151.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2023-10-18T15:54:06.202 INFO:teuthology.orchestra.run.smithi151.stdout:ANSI_COLOR="0;31" 2023-10-18T15:54:06.202 INFO:teuthology.orchestra.run.smithi151.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2023-10-18T15:54:06.202 INFO:teuthology.orchestra.run.smithi151.stdout:HOME_URL="https://www.redhat.com/" 2023-10-18T15:54:06.203 INFO:teuthology.orchestra.run.smithi151.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-10-18T15:54:06.203 INFO:teuthology.orchestra.run.smithi151.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-18T15:54:06.203 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-10-18T15:54:06.203 INFO:teuthology.orchestra.run.smithi151.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-18T15:54:06.203 INFO:teuthology.orchestra.run.smithi151.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2023-10-18T15:54:06.204 INFO:teuthology.orchestra.run.smithi151.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-10-18T15:54:06.204 INFO:teuthology.orchestra.run.smithi151.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2023-10-18T15:54:06.204 INFO:teuthology.lock.ops:Updating smithi151.front.sepia.ceph.com on lock server 2023-10-18T15:54:06.226 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-18T15:54:06.236 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-18T15:54:06.301 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-18T15:54:06.301 DEBUG:teuthology.orchestra.run.smithi088:> test '!' -e /home/ubuntu/cephtest 2023-10-18T15:54:06.305 DEBUG:teuthology.orchestra.run.smithi151:> test '!' -e /home/ubuntu/cephtest 2023-10-18T15:54:06.320 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-18T15:54:06.329 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-18T15:54:06.330 DEBUG:teuthology.orchestra.run.smithi088:> test -z $(ls -A /var/lib/ceph) 2023-10-18T15:54:06.361 DEBUG:teuthology.orchestra.run.smithi151:> test -z $(ls -A /var/lib/ceph) 2023-10-18T15:54:06.374 INFO:teuthology.orchestra.run.smithi088.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2023-10-18T15:54:06.377 INFO:teuthology.orchestra.run.smithi151.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2023-10-18T15:54:06.378 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-18T15:54:06.434 INFO:teuthology.run_tasks:Running task kernel... 2023-10-18T15:54:06.450 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': True, 'sha1': '47672119149b103ec7cec2cc65c063ccbb001ac7'} 2023-10-18T15:54:06.451 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2023-10-18T15:54:06.451 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2023-10-18T15:54:06.451 DEBUG:teuthology.orchestra.run.smithi088:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-18T15:54:06.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T15:54:06.468 DEBUG:teuthology.orchestra.run.smithi088:> uname -r 2023-10-18T15:54:06.524 INFO:teuthology.orchestra.run.smithi088.stdout:4.18.0-305.25.1.el8_4.x86_64 2023-10-18T15:54:06.525 INFO:teuthology.task.kernel:Running kernel on smithi088: 4.18.0-305.25.1.el8_4.x86_64 2023-10-18T15:54:06.525 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum install -y kernel 2023-10-18T15:54:07.989 INFO:teuthology.orchestra.run.smithi088.stdout:Updating Subscription Management repositories. 2023-10-18T15:54:07.990 INFO:teuthology.orchestra.run.smithi088.stdout:Unable to read consumer identity 2023-10-18T15:54:09.370 INFO:teuthology.orchestra.run.smithi088.stdout:Copr repo for python3-asyncssh owned by ceph 28 kB/s | 3.5 kB 00:00 2023-10-18T15:54:09.897 INFO:teuthology.orchestra.run.smithi088.stdout:Extra Packages for Enterprise Linux 33 MB/s | 16 MB 00:00 2023-10-18T15:54:13.239 INFO:teuthology.orchestra.run.smithi088.stdout:lab-extras 4.9 MB/s | 24 kB 00:00 2023-10-18T15:54:14.016 INFO:teuthology.orchestra.run.smithi088.stdout:Package kernel-4.18.0-80.el8.x86_64 is already installed. 2023-10-18T15:54:14.016 INFO:teuthology.orchestra.run.smithi088.stdout:Package kernel-4.18.0-425.3.1.el8.x86_64 is already installed. 2023-10-18T15:54:14.016 INFO:teuthology.orchestra.run.smithi088.stdout:Package kernel-4.18.0-305.25.1.el8_4.x86_64 is already installed. 2023-10-18T15:54:14.162 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2023-10-18T15:54:14.163 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2023-10-18T15:54:14.163 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2023-10-18T15:54:14.299 DEBUG:teuthology.orchestra.run.smithi088:> echo no | sudo yum reinstall kernel || true 2023-10-18T15:54:14.644 INFO:teuthology.orchestra.run.smithi088.stdout:Updating Subscription Management repositories. 2023-10-18T15:54:14.644 INFO:teuthology.orchestra.run.smithi088.stdout:Unable to read consumer identity 2023-10-18T15:54:14.939 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:00:01 ago on Wed 18 Oct 2023 03:54:13 PM UTC. 2023-10-18T15:54:15.080 INFO:teuthology.orchestra.run.smithi088.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-10-18T15:54:15.080 INFO:teuthology.orchestra.run.smithi088.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-18T15:54:15.080 INFO:teuthology.orchestra.run.smithi088.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-18T15:54:15.106 INFO:teuthology.orchestra.run.smithi088.stderr:Error: No packages marked for reinstall. 2023-10-18T15:54:15.155 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum reinstall -y kernel || true 2023-10-18T15:54:15.498 INFO:teuthology.orchestra.run.smithi088.stdout:Updating Subscription Management repositories. 2023-10-18T15:54:15.498 INFO:teuthology.orchestra.run.smithi088.stdout:Unable to read consumer identity 2023-10-18T15:54:15.780 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:00:02 ago on Wed 18 Oct 2023 03:54:13 PM UTC. 2023-10-18T15:54:15.954 INFO:teuthology.orchestra.run.smithi088.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-10-18T15:54:15.954 INFO:teuthology.orchestra.run.smithi088.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-18T15:54:15.954 INFO:teuthology.orchestra.run.smithi088.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-18T15:54:15.964 INFO:teuthology.orchestra.run.smithi088.stderr:Error: No packages marked for reinstall. 2023-10-18T15:54:16.013 DEBUG:teuthology.orchestra.run.smithi088:> rpm -q kernel | sort -rV | head -n 1 2023-10-18T15:54:16.138 INFO:teuthology.orchestra.run.smithi088.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-10-18T15:54:16.139 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-10-18T15:54:16.139 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-305.25.1.el8_4.x86_64 Expected: 4.18.0-425.3.1.el8.x86_64 2023-10-18T15:54:16.139 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-18T15:54:16.140 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-18T15:54:16.140 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2023-10-18T15:54:16.140 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi088.front.sepia.ceph.com, path=None, version=distro) 2023-10-18T15:54:16.140 DEBUG:teuthology.orchestra.run.smithi088:> rpm -q kernel | sort -rV | head -n 1 2023-10-18T15:54:16.265 INFO:teuthology.orchestra.run.smithi088.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-10-18T15:54:16.265 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-10-18T15:54:16.266 DEBUG:teuthology.orchestra.run.smithi088:> sudo rpm -qi grub2-tools 2023-10-18T15:54:16.292 INFO:teuthology.orchestra.run.smithi088.stdout:Name : grub2-tools 2023-10-18T15:54:16.292 INFO:teuthology.orchestra.run.smithi088.stdout:Epoch : 1 2023-10-18T15:54:16.292 INFO:teuthology.orchestra.run.smithi088.stdout:Version : 2.02 2023-10-18T15:54:16.293 INFO:teuthology.orchestra.run.smithi088.stdout:Release : 99.el8_4.1 2023-10-18T15:54:16.293 INFO:teuthology.orchestra.run.smithi088.stdout:Architecture: x86_64 2023-10-18T15:54:16.293 INFO:teuthology.orchestra.run.smithi088.stdout:Install Date: Wed 21 Dec 2022 01:02:32 PM UTC 2023-10-18T15:54:16.293 INFO:teuthology.orchestra.run.smithi088.stdout:Group : System Environment/Base 2023-10-18T15:54:16.293 INFO:teuthology.orchestra.run.smithi088.stdout:Size : 9264631 2023-10-18T15:54:16.294 INFO:teuthology.orchestra.run.smithi088.stdout:License : GPLv3+ 2023-10-18T15:54:16.294 INFO:teuthology.orchestra.run.smithi088.stdout:Signature : RSA/SHA256, Thu 09 Sep 2021 09:41:55 AM UTC, Key ID 199e2f91fd431d51 2023-10-18T15:54:16.294 INFO:teuthology.orchestra.run.smithi088.stdout:Source RPM : grub2-2.02-99.el8_4.1.src.rpm 2023-10-18T15:54:16.294 INFO:teuthology.orchestra.run.smithi088.stdout:Build Date : Tue 18 May 2021 07:46:34 AM UTC 2023-10-18T15:54:16.294 INFO:teuthology.orchestra.run.smithi088.stdout:Build Host : x86-vm-07.build.eng.bos.redhat.com 2023-10-18T15:54:16.295 INFO:teuthology.orchestra.run.smithi088.stdout:Relocations : (not relocatable) 2023-10-18T15:54:16.295 INFO:teuthology.orchestra.run.smithi088.stdout:Packager : Red Hat, Inc. 2023-10-18T15:54:16.295 INFO:teuthology.orchestra.run.smithi088.stdout:Vendor : Red Hat, Inc. 2023-10-18T15:54:16.295 INFO:teuthology.orchestra.run.smithi088.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-18T15:54:16.295 INFO:teuthology.orchestra.run.smithi088.stdout:Summary : Support tools for GRUB. 2023-10-18T15:54:16.296 INFO:teuthology.orchestra.run.smithi088.stdout:Description : 2023-10-18T15:54:16.296 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T15:54:16.296 INFO:teuthology.orchestra.run.smithi088.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-18T15:54:16.296 INFO:teuthology.orchestra.run.smithi088.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-18T15:54:16.296 INFO:teuthology.orchestra.run.smithi088.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-18T15:54:16.297 INFO:teuthology.orchestra.run.smithi088.stdout:hardware devices. 2023-10-18T15:54:16.297 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T15:54:16.297 INFO:teuthology.orchestra.run.smithi088.stdout:This subpackage provides tools for support of all platforms. 2023-10-18T15:54:16.298 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-18T15:54:16.298 INFO:teuthology.task.kernel:Updating grub on smithi088 to boot 4.18.0-425.3.1.el8.x86_64 2023-10-18T15:54:16.301 DEBUG:teuthology.orchestra.run.smithi088:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-18T15:54:17.714 INFO:teuthology.orchestra.run.smithi088.stderr:Generating grub configuration file ... 2023-10-18T15:54:19.787 INFO:teuthology.orchestra.run.smithi088.stderr:done 2023-10-18T15:54:19.788 DEBUG:teuthology.orchestra.run.smithi088:> mktemp 2023-10-18T15:54:19.803 INFO:teuthology.orchestra.run.smithi088.stdout:/tmp/tmp.cmFPfzmVex 2023-10-18T15:54:19.803 DEBUG:teuthology.orchestra.run.smithi088:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.cmFPfzmVex 2023-10-18T15:54:19.876 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 0666 /tmp/tmp.cmFPfzmVex 2023-10-18T15:54:20.027 DEBUG:teuthology.orchestra.remote:smithi088:/tmp/tmp.cmFPfzmVex is 6KB 2023-10-18T15:54:20.077 DEBUG:teuthology.orchestra.run.smithi088:> rm -fr /tmp/tmp.cmFPfzmVex 2023-10-18T15:54:20.093 DEBUG:teuthology.orchestra.run.smithi088:> sudo /bin/ls /boot/loader/entries || true 2023-10-18T15:54:20.160 INFO:teuthology.orchestra.run.smithi088.stdout:298e228f3b5e401e8ac0f5ae159a2e06-0-rescue.conf 2023-10-18T15:54:20.161 INFO:teuthology.orchestra.run.smithi088.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-305.25.1.el8_4.x86_64.conf 2023-10-18T15:54:20.161 INFO:teuthology.orchestra.run.smithi088.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64.conf 2023-10-18T15:54:20.161 INFO:teuthology.orchestra.run.smithi088.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-80.el8.x86_64.conf 2023-10-18T15:54:20.163 DEBUG:teuthology.orchestra.run.smithi088:> sudo grub2-set-default 298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64 2023-10-18T15:54:20.320 DEBUG:teuthology.orchestra.run.smithi088:> sudo shutdown -r now 2023-10-18T15:54:50.323 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2023-10-18T15:54:50.324 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2023-10-18T15:54:50.325 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T15:55:24.922 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.88 2023-10-18T15:55:33.924 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2023-10-18T15:55:33.925 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T15:55:36.986 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.88 2023-10-18T15:55:48.988 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2023-10-18T15:55:48.988 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T15:56:21.808 DEBUG:teuthology.orchestra.run.smithi088:> true 2023-10-18T15:56:22.735 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi088.front.sepia.ceph.com' 2023-10-18T15:56:22.736 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "4.18.0-425.3.1.el8.x86_64"... 2023-10-18T15:56:22.737 DEBUG:teuthology.orchestra.run.smithi088:> uname -r 2023-10-18T15:56:22.790 INFO:teuthology.orchestra.run.smithi088.stdout:4.18.0-425.3.1.el8.x86_64 2023-10-18T15:56:22.791 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-425.3.1.el8.x86_64 vs 4.18.0-425.3.1.el8.x86_64 2023-10-18T15:56:22.791 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-18T15:56:22.791 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2023-10-18T15:56:23.793 DEBUG:teuthology.parallel:result is None 2023-10-18T15:56:23.794 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-18T15:56:23.801 INFO:teuthology.task.internal:Creating test directory... 2023-10-18T15:56:23.802 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-18T15:56:23.804 DEBUG:teuthology.orchestra.run.smithi151:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-18T15:56:23.822 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-18T15:56:23.836 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-18T15:56:23.869 INFO:teuthology.task.internal:Creating archive directory... 2023-10-18T15:56:23.870 DEBUG:teuthology.orchestra.run.smithi088:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-18T15:56:23.872 DEBUG:teuthology.orchestra.run.smithi151:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-18T15:56:23.902 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-18T15:56:23.909 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-18T15:56:23.909 DEBUG:teuthology.orchestra.run.smithi151:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-18T15:56:23.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T15:56:23.953 DEBUG:teuthology.orchestra.run.smithi088:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-18T15:56:23.956 DEBUG:teuthology.orchestra.run.smithi151:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-18T15:56:24.033 INFO:teuthology.orchestra.run.smithi151.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-18T15:56:24.034 INFO:teuthology.orchestra.run.smithi088.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-18T15:56:24.047 INFO:teuthology.orchestra.run.smithi088.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-18T15:56:24.081 INFO:teuthology.orchestra.run.smithi151.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-18T15:56:24.083 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-18T15:56:24.091 INFO:teuthology.task.internal:Configuring sudo... 2023-10-18T15:56:24.091 DEBUG:teuthology.orchestra.run.smithi088:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-18T15:56:24.093 DEBUG:teuthology.orchestra.run.smithi151:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-18T15:56:24.151 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-18T15:56:24.161 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-18T15:56:24.162 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-18T15:56:24.164 DEBUG:teuthology.orchestra.run.smithi151:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-18T15:56:24.205 DEBUG:teuthology.orchestra.run.smithi088:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-18T15:56:24.258 DEBUG:teuthology.orchestra.run.smithi088:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-18T15:56:24.315 DEBUG:teuthology.orchestra.run.smithi088:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-18T15:56:24.395 DEBUG:teuthology.orchestra.run.smithi088:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-18T15:56:24.435 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-18T15:56:24.436 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-18T15:56:24.499 DEBUG:teuthology.orchestra.run.smithi151:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-18T15:56:24.541 DEBUG:teuthology.orchestra.run.smithi151:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-18T15:56:24.585 DEBUG:teuthology.orchestra.run.smithi151:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-18T15:56:24.667 DEBUG:teuthology.orchestra.run.smithi151:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-18T15:56:24.708 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-10-18T15:56:24.708 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-18T15:56:24.772 DEBUG:teuthology.orchestra.run.smithi088:> sudo service rsyslog restart 2023-10-18T15:56:24.774 DEBUG:teuthology.orchestra.run.smithi151:> sudo service rsyslog restart 2023-10-18T15:56:24.823 INFO:teuthology.orchestra.run.smithi088.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-18T15:56:24.877 INFO:teuthology.orchestra.run.smithi151.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-18T15:56:25.259 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-18T15:56:25.269 INFO:teuthology.task.internal:Starting timer... 2023-10-18T15:56:25.269 INFO:teuthology.run_tasks:Running task pcp... 2023-10-18T15:56:25.314 INFO:teuthology.run_tasks:Running task selinux... 2023-10-18T15:56:25.346 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-10-18T15:56:25.347 DEBUG:teuthology.orchestra.run.smithi088:> sudo service auditd rotate 2023-10-18T15:56:25.411 INFO:teuthology.orchestra.run.smithi088.stdout:Rotating logs: [ OK ] 2023-10-18T15:56:25.413 DEBUG:teuthology.orchestra.run.smithi151:> sudo service auditd rotate 2023-10-18T15:56:25.475 INFO:teuthology.orchestra.run.smithi151.stdout:Rotating logs: [ OK ] 2023-10-18T15:56:25.477 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-18T15:56:25.477 DEBUG:teuthology.orchestra.run.smithi088:> /usr/sbin/getenforce 2023-10-18T15:56:25.492 INFO:teuthology.orchestra.run.smithi088.stdout:Permissive 2023-10-18T15:56:25.492 DEBUG:teuthology.orchestra.run.smithi151:> /usr/sbin/getenforce 2023-10-18T15:56:25.512 INFO:teuthology.orchestra.run.smithi151.stdout:Permissive 2023-10-18T15:56:25.512 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi088.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi151.front.sepia.ceph.com': 'permissive'} 2023-10-18T15:56:25.512 DEBUG:teuthology.orchestra.run.smithi088:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-18T15:56:25.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T15:56:25.557 DEBUG:teuthology.orchestra.run.smithi151:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-18T15:56:25.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T15:56:25.583 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-18T15:56:25.584 DEBUG:teuthology.orchestra.run.smithi088:> sudo /usr/sbin/setenforce permissive 2023-10-18T15:56:25.629 DEBUG:teuthology.orchestra.run.smithi151:> sudo /usr/sbin/setenforce permissive 2023-10-18T15:56:25.660 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-18T15:56:25.669 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-10-18T15:56:25.675 INFO:teuthology.repo_utils:Fetching main from origin 2023-10-18T15:56:25.796 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-18T15:56:25.812 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-18T15:56:25.813 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi088.front.sepia.ceph.com,smithi151.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-18T16:04:33.340 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi088.front.sepia.ceph.com'), Remote(name='ubuntu@smithi151.front.sepia.ceph.com')] 2023-10-18T16:04:33.342 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2023-10-18T16:04:33.344 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T16:04:33.414 DEBUG:teuthology.orchestra.run.smithi088:> true 2023-10-18T16:04:33.481 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi088.front.sepia.ceph.com' 2023-10-18T16:04:33.482 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-10-18T16:04:33.482 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T16:04:33.552 DEBUG:teuthology.orchestra.run.smithi151:> true 2023-10-18T16:04:33.618 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-10-18T16:04:33.619 INFO:teuthology.run_tasks:Running task clock... 2023-10-18T16:04:33.631 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-10-18T16:04:33.632 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-18T16:04:33.632 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-18T16:04:33.635 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-18T16:04:33.635 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-18T16:04:33.662 INFO:teuthology.orchestra.run.smithi088.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-18T16:04:33.677 INFO:teuthology.orchestra.run.smithi088.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-18T16:04:33.687 INFO:teuthology.orchestra.run.smithi151.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-18T16:04:33.701 INFO:teuthology.orchestra.run.smithi151.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-18T16:04:33.711 INFO:teuthology.orchestra.run.smithi088.stderr:sudo: ntpd: command not found 2023-10-18T16:04:33.721 INFO:teuthology.orchestra.run.smithi088.stdout:506 Cannot talk to daemon 2023-10-18T16:04:33.734 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: ntpd: command not found 2023-10-18T16:04:33.735 INFO:teuthology.orchestra.run.smithi088.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-18T16:04:33.744 INFO:teuthology.orchestra.run.smithi151.stdout:506 Cannot talk to daemon 2023-10-18T16:04:33.749 INFO:teuthology.orchestra.run.smithi088.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-18T16:04:33.757 INFO:teuthology.orchestra.run.smithi151.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-18T16:04:33.770 INFO:teuthology.orchestra.run.smithi151.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-18T16:04:33.825 INFO:teuthology.orchestra.run.smithi088.stderr:bash: ntpq: command not found 2023-10-18T16:04:33.828 INFO:teuthology.orchestra.run.smithi088.stdout:210 Number of sources = 4 2023-10-18T16:04:33.829 INFO:teuthology.orchestra.run.smithi088.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-18T16:04:33.829 INFO:teuthology.orchestra.run.smithi088.stdout:=============================================================================== 2023-10-18T16:04:33.830 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-18T16:04:33.830 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-18T16:04:33.830 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-18T16:04:33.830 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-18T16:04:33.836 INFO:teuthology.orchestra.run.smithi151.stderr:bash: ntpq: command not found 2023-10-18T16:04:33.839 INFO:teuthology.orchestra.run.smithi151.stdout:210 Number of sources = 4 2023-10-18T16:04:33.839 INFO:teuthology.orchestra.run.smithi151.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-18T16:04:33.839 INFO:teuthology.orchestra.run.smithi151.stdout:=============================================================================== 2023-10-18T16:04:33.839 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-18T16:04:33.839 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-18T16:04:33.840 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-18T16:04:33.840 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-18T16:04:33.841 INFO:teuthology.run_tasks:Running task install... 2023-10-18T16:04:33.850 DEBUG:teuthology.task.install:project ceph 2023-10-18T16:04:33.850 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5'}} 2023-10-18T16:04:33.851 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'flavor': 'default', 'sha1': 'b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5'} 2023-10-18T16:04:33.851 INFO:teuthology.task.install:Using flavor: default 2023-10-18T16:04:33.857 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-18T16:04:33.857 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-10-18T16:04:33.858 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5', 'tag': None, 'wait_for_package': False} 2023-10-18T16:04:33.858 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5 2023-10-18T16:04:33.861 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5', 'tag': None, 'wait_for_package': False} 2023-10-18T16:04:33.862 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5 2023-10-18T16:04:34.019 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-pacific-20231016.091648/b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/centos/8/flavors/default/ 2023-10-18T16:04:34.019 INFO:teuthology.task.install.rpm:Package version is 16.2.14-100.gb4ddd52c 2023-10-18T16:04:34.029 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-pacific-20231016.091648/b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/centos/8/flavors/default/ 2023-10-18T16:04:34.029 INFO:teuthology.task.install.rpm:Package version is 16.2.14-100.gb4ddd52c 2023-10-18T16:04:34.139 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-pacific-20231016.091648/b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-pacific-20231016.091648/b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-pacific-20231016.091648/b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-18T16:04:34.140 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-10-18T16:04:34.140 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-18T16:04:34.165 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel on remote rpm x86_64 2023-10-18T16:04:34.166 DEBUG:teuthology.orchestra.run.smithi151:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-18T16:04:34.191 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-pacific-20231016.091648/b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-pacific-20231016.091648/b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-pacific-20231016.091648/b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-18T16:04:34.192 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-18T16:04:34.192 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-18T16:04:34.218 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel on remote rpm x86_64 2023-10-18T16:04:34.219 DEBUG:teuthology.orchestra.run.smithi088:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-18T16:04:34.229 DEBUG:teuthology.orchestra.run.smithi151:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-18T16:04:34.283 DEBUG:teuthology.orchestra.run.smithi088:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-18T16:04:34.302 DEBUG:teuthology.orchestra.run.smithi151:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-18T16:04:34.357 DEBUG:teuthology.orchestra.run.smithi088:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-18T16:04:34.365 INFO:teuthology.orchestra.run.smithi151.stdout:check_obsoletes = 1 2023-10-18T16:04:34.366 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum clean all 2023-10-18T16:04:34.422 INFO:teuthology.orchestra.run.smithi088.stdout:check_obsoletes = 1 2023-10-18T16:04:34.423 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum clean all 2023-10-18T16:04:34.751 INFO:teuthology.orchestra.run.smithi151.stdout:Updating Subscription Management repositories. 2023-10-18T16:04:34.812 INFO:teuthology.orchestra.run.smithi088.stdout:Updating Subscription Management repositories. 2023-10-18T16:04:35.040 INFO:teuthology.orchestra.run.smithi151.stdout:44 files removed 2023-10-18T16:04:35.090 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel 2023-10-18T16:04:35.091 INFO:teuthology.orchestra.run.smithi088.stdout:44 files removed 2023-10-18T16:04:35.141 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel 2023-10-18T16:04:35.435 INFO:teuthology.orchestra.run.smithi151.stdout:Updating Subscription Management repositories. 2023-10-18T16:04:35.490 INFO:teuthology.orchestra.run.smithi088.stdout:Updating Subscription Management repositories. 2023-10-18T16:04:36.244 INFO:teuthology.orchestra.run.smithi151.stdout:ceph packages for x86_64 201 kB/s | 76 kB 00:00 2023-10-18T16:04:36.299 INFO:teuthology.orchestra.run.smithi088.stdout:ceph packages for x86_64 199 kB/s | 76 kB 00:00 2023-10-18T16:04:36.587 INFO:teuthology.orchestra.run.smithi088.stdout:ceph noarch packages 62 kB/s | 15 kB 00:00 2023-10-18T16:04:36.593 INFO:teuthology.orchestra.run.smithi151.stdout:ceph noarch packages 50 kB/s | 15 kB 00:00 2023-10-18T16:04:36.877 INFO:teuthology.orchestra.run.smithi151.stdout:ceph source packages 5.7 kB/s | 1.4 kB 00:00 2023-10-18T16:04:36.929 INFO:teuthology.orchestra.run.smithi088.stdout:ceph source packages 4.6 kB/s | 1.4 kB 00:00 2023-10-18T16:04:37.000 INFO:teuthology.orchestra.run.smithi151.stdout:Copr repo for python3-asyncssh owned by ceph 36 kB/s | 3.5 kB 00:00 2023-10-18T16:04:37.045 INFO:teuthology.orchestra.run.smithi088.stdout:Copr repo for python3-asyncssh owned by ceph 40 kB/s | 3.5 kB 00:00 2023-10-18T16:04:38.167 INFO:teuthology.orchestra.run.smithi088.stdout:Extra Packages for Enterprise Linux 15 MB/s | 16 MB 00:01 2023-10-18T16:04:41.473 INFO:teuthology.orchestra.run.smithi088.stdout:lab-extras 1.7 MB/s | 24 kB 00:00 2023-10-18T16:04:41.855 INFO:teuthology.orchestra.run.smithi088.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 127 MB/s | 38 MB 00:00 2023-10-18T16:04:43.233 INFO:teuthology.orchestra.run.smithi151.stdout:Extra Packages for Enterprise Linux 2.6 MB/s | 16 MB 00:06 2023-10-18T16:04:46.519 INFO:teuthology.orchestra.run.smithi151.stdout:lab-extras 4.0 MB/s | 24 kB 00:00 2023-10-18T16:04:46.907 INFO:teuthology.orchestra.run.smithi151.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 125 MB/s | 38 MB 00:00 2023-10-18T16:04:47.285 INFO:teuthology.orchestra.run.smithi088.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 124 MB/s | 33 MB 00:00 2023-10-18T16:04:52.383 INFO:teuthology.orchestra.run.smithi151.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 124 MB/s | 33 MB 00:00 2023-10-18T16:04:55.162 INFO:teuthology.orchestra.run.smithi088.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-10-18T16:04:55.163 INFO:teuthology.orchestra.run.smithi088.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-10-18T16:04:55.174 INFO:teuthology.orchestra.run.smithi088.stdout:Package NetworkManager-1:1.30.0-13.el8_4.x86_64 is already installed. 2023-10-18T16:04:55.174 INFO:teuthology.orchestra.run.smithi088.stdout:Package iproute-5.9.0-4.el8.x86_64 is already installed. 2023-10-18T16:04:55.176 INFO:teuthology.orchestra.run.smithi088.stdout:Package util-linux-2.32.1-27.el8.x86_64 is already installed. 2023-10-18T16:04:55.176 INFO:teuthology.orchestra.run.smithi088.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-10-18T16:04:55.178 INFO:teuthology.orchestra.run.smithi088.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-10-18T16:04:55.180 INFO:teuthology.orchestra.run.smithi088.stdout:Package xfsdump-3.1.8-2.el8.x86_64 is already installed. 2023-10-18T16:04:55.181 INFO:teuthology.orchestra.run.smithi088.stdout:Package xfsprogs-5.0.0-8.el8.x86_64 is already installed. 2023-10-18T16:04:55.664 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2023-10-18T16:04:55.675 INFO:teuthology.orchestra.run.smithi088.stdout:====================================================================================================================== 2023-10-18T16:04:55.676 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2023-10-18T16:04:55.676 INFO:teuthology.orchestra.run.smithi088.stdout:====================================================================================================================== 2023-10-18T16:04:55.676 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2023-10-18T16:04:55.676 INFO:teuthology.orchestra.run.smithi088.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-10-18T16:04:55.677 INFO:teuthology.orchestra.run.smithi088.stdout: ceph x86_64 2:16.2.14-100.gb4ddd52c.el8 ceph 6.4 k 2023-10-18T16:04:55.677 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-base x86_64 2:16.2.14-100.gb4ddd52c.el8 ceph 6.1 M 2023-10-18T16:04:55.677 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-fuse x86_64 2:16.2.14-100.gb4ddd52c.el8 ceph 912 k 2023-10-18T16:04:55.677 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-immutable-object-cache x86_64 2:16.2.14-100.gb4ddd52c.el8 ceph 161 k 2023-10-18T16:04:55.677 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr x86_64 2:16.2.14-100.gb4ddd52c.el8 ceph 1.6 M 2023-10-18T16:04:55.677 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-cephadm noarch 2:16.2.14-100.gb4ddd52c.el8 ceph-noarch 116 k 2023-10-18T16:04:55.678 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-dashboard noarch 2:16.2.14-100.gb4ddd52c.el8 ceph-noarch 3.1 M 2023-10-18T16:04:55.678 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.14-100.gb4ddd52c.el8 ceph-noarch 8.6 M 2023-10-18T16:04:55.678 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-rook noarch 2:16.2.14-100.gb4ddd52c.el8 ceph-noarch 27 k 2023-10-18T16:04:55.678 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-radosgw x86_64 2:16.2.14-100.gb4ddd52c.el8 ceph 12 M 2023-10-18T16:04:55.678 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-test x86_64 2:16.2.14-100.gb4ddd52c.el8 ceph 52 M 2023-10-18T16:04:55.679 INFO:teuthology.orchestra.run.smithi088.stdout: cephadm noarch 2:16.2.14-100.gb4ddd52c.el8 ceph-noarch 84 k 2023-10-18T16:04:55.679 INFO:teuthology.orchestra.run.smithi088.stdout: cephfs-mirror x86_64 2:16.2.14-100.gb4ddd52c.el8 ceph 257 k 2023-10-18T16:04:55.679 INFO:teuthology.orchestra.run.smithi088.stdout: cephfs-top noarch 2:16.2.14-100.gb4ddd52c.el8 ceph-noarch 19 k 2023-10-18T16:04:55.679 INFO:teuthology.orchestra.run.smithi088.stdout: elfutils-libelf-devel x86_64 0.182-3.el8 rhel-8-for-x86_64-baseos-rpms 59 k 2023-10-18T16:04:55.679 INFO:teuthology.orchestra.run.smithi088.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-10-18T16:04:55.680 INFO:teuthology.orchestra.run.smithi088.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-10-18T16:04:55.680 INFO:teuthology.orchestra.run.smithi088.stdout: libcephfs-devel x86_64 2:16.2.14-100.gb4ddd52c.el8 ceph 25 k 2023-10-18T16:04:55.680 INFO:teuthology.orchestra.run.smithi088.stdout: libcephfs2 x86_64 2:16.2.14-100.gb4ddd52c.el8 ceph 825 k 2023-10-18T16:04:55.680 INFO:teuthology.orchestra.run.smithi088.stdout: librados-devel x86_64 2:16.2.14-100.gb4ddd52c.el8 ceph 131 k 2023-10-18T16:04:55.680 INFO:teuthology.orchestra.run.smithi088.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-10-18T16:04:55.680 INFO:teuthology.orchestra.run.smithi088.stdout: libuuid-devel x86_64 2.32.1-27.el8 rhel-8-for-x86_64-baseos-rpms 97 k 2023-10-18T16:04:55.681 INFO:teuthology.orchestra.run.smithi088.stdout: openssl-devel x86_64 1:1.1.1g-15.el8_3 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-10-18T16:04:55.681 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cephfs x86_64 2:16.2.14-100.gb4ddd52c.el8 ceph 215 k 2023-10-18T16:04:55.681 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rados x86_64 2:16.2.14-100.gb4ddd52c.el8 ceph 390 k 2023-10-18T16:04:55.681 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rbd x86_64 2:16.2.14-100.gb4ddd52c.el8 ceph 369 k 2023-10-18T16:04:55.681 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rgw x86_64 2:16.2.14-100.gb4ddd52c.el8 ceph 114 k 2023-10-18T16:04:55.681 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-fuse x86_64 2:16.2.14-100.gb4ddd52c.el8 ceph 88 k 2023-10-18T16:04:55.682 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-mirror x86_64 2:16.2.14-100.gb4ddd52c.el8 ceph 3.7 M 2023-10-18T16:04:55.682 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-nbd x86_64 2:16.2.14-100.gb4ddd52c.el8 ceph 173 k 2023-10-18T16:04:55.682 INFO:teuthology.orchestra.run.smithi088.stdout: xfsprogs-devel x86_64 5.0.0-8.el8 rhel-8-for-x86_64-baseos-rpms 70 k 2023-10-18T16:04:55.682 INFO:teuthology.orchestra.run.smithi088.stdout:Upgrading: 2023-10-18T16:04:55.682 INFO:teuthology.orchestra.run.smithi088.stdout: librados2 x86_64 2:16.2.14-100.gb4ddd52c.el8 ceph 3.7 M 2023-10-18T16:04:55.683 INFO:teuthology.orchestra.run.smithi088.stdout: librbd1 x86_64 2:16.2.14-100.gb4ddd52c.el8 ceph 4.2 M 2023-10-18T16:04:55.683 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2023-10-18T16:04:55.683 INFO:teuthology.orchestra.run.smithi088.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-10-18T16:04:55.683 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-common x86_64 2:16.2.14-100.gb4ddd52c.el8 ceph 24 M 2023-10-18T16:04:55.683 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-grafana-dashboards noarch 2:16.2.14-100.gb4ddd52c.el8 ceph-noarch 24 k 2023-10-18T16:04:55.684 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mds x86_64 2:16.2.14-100.gb4ddd52c.el8 ceph 2.4 M 2023-10-18T16:04:55.684 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-modules-core noarch 2:16.2.14-100.gb4ddd52c.el8 ceph-noarch 244 k 2023-10-18T16:04:55.684 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mon x86_64 2:16.2.14-100.gb4ddd52c.el8 ceph 5.0 M 2023-10-18T16:04:55.684 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-osd x86_64 2:16.2.14-100.gb4ddd52c.el8 ceph 18 M 2023-10-18T16:04:55.684 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-prometheus-alerts noarch 2:16.2.14-100.gb4ddd52c.el8 ceph-noarch 15 k 2023-10-18T16:04:55.685 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-selinux x86_64 2:16.2.14-100.gb4ddd52c.el8 ceph 26 k 2023-10-18T16:04:55.685 INFO:teuthology.orchestra.run.smithi088.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-10-18T16:04:55.685 INFO:teuthology.orchestra.run.smithi088.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-10-18T16:04:55.685 INFO:teuthology.orchestra.run.smithi088.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2023-10-18T16:04:55.685 INFO:teuthology.orchestra.run.smithi088.stdout: keyutils-libs-devel x86_64 1.5.10-6.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-10-18T16:04:55.685 INFO:teuthology.orchestra.run.smithi088.stdout: krb5-devel x86_64 1.18.2-8.3.el8_4 rhel-8-for-x86_64-baseos-rpms 559 k 2023-10-18T16:04:55.686 INFO:teuthology.orchestra.run.smithi088.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2023-10-18T16:04:55.686 INFO:teuthology.orchestra.run.smithi088.stdout: libcephsqlite x86_64 2:16.2.14-100.gb4ddd52c.el8 ceph 174 k 2023-10-18T16:04:55.686 INFO:teuthology.orchestra.run.smithi088.stdout: libcom_err-devel x86_64 1.45.6-1.el8 rhel-8-for-x86_64-baseos-rpms 38 k 2023-10-18T16:04:55.686 INFO:teuthology.orchestra.run.smithi088.stdout: libgfortran x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2023-10-18T16:04:55.686 INFO:teuthology.orchestra.run.smithi088.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-10-18T16:04:55.687 INFO:teuthology.orchestra.run.smithi088.stdout: libquadmath x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2023-10-18T16:04:55.687 INFO:teuthology.orchestra.run.smithi088.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-10-18T16:04:55.687 INFO:teuthology.orchestra.run.smithi088.stdout: libradosstriper1 x86_64 2:16.2.14-100.gb4ddd52c.el8 ceph 494 k 2023-10-18T16:04:55.687 INFO:teuthology.orchestra.run.smithi088.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2023-10-18T16:04:55.687 INFO:teuthology.orchestra.run.smithi088.stdout: librgw2 x86_64 2:16.2.14-100.gb4ddd52c.el8 ceph 3.8 M 2023-10-18T16:04:55.688 INFO:teuthology.orchestra.run.smithi088.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-10-18T16:04:55.688 INFO:teuthology.orchestra.run.smithi088.stdout: libsepol-devel x86_64 2.9-2.el8 rhel-8-for-x86_64-baseos-rpms 86 k 2023-10-18T16:04:55.688 INFO:teuthology.orchestra.run.smithi088.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-10-18T16:04:55.688 INFO:teuthology.orchestra.run.smithi088.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-10-18T16:04:55.688 INFO:teuthology.orchestra.run.smithi088.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-10-18T16:04:55.688 INFO:teuthology.orchestra.run.smithi088.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-10-18T16:04:55.689 INFO:teuthology.orchestra.run.smithi088.stdout: openblas x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.6 M 2023-10-18T16:04:55.689 INFO:teuthology.orchestra.run.smithi088.stdout: openblas-threads x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-10-18T16:04:55.689 INFO:teuthology.orchestra.run.smithi088.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-10-18T16:04:55.689 INFO:teuthology.orchestra.run.smithi088.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-10-18T16:04:55.689 INFO:teuthology.orchestra.run.smithi088.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-10-18T16:04:55.689 INFO:teuthology.orchestra.run.smithi088.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2023-10-18T16:04:55.690 INFO:teuthology.orchestra.run.smithi088.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-10-18T16:04:55.690 INFO:teuthology.orchestra.run.smithi088.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-10-18T16:04:55.690 INFO:teuthology.orchestra.run.smithi088.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-10-18T16:04:55.690 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-10-18T16:04:55.690 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ceph-argparse x86_64 2:16.2.14-100.gb4ddd52c.el8 ceph 45 k 2023-10-18T16:04:55.691 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ceph-common x86_64 2:16.2.14-100.gb4ddd52c.el8 ceph 99 k 2023-10-18T16:04:55.691 INFO:teuthology.orchestra.run.smithi088.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-10-18T16:04:55.691 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-10-18T16:04:55.691 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-10-18T16:04:55.691 INFO:teuthology.orchestra.run.smithi088.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2023-10-18T16:04:55.692 INFO:teuthology.orchestra.run.smithi088.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-10-18T16:04:55.692 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-10-18T16:04:55.692 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-10-18T16:04:55.692 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2023-10-18T16:04:55.692 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-10-18T16:04:55.692 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-10-18T16:04:55.693 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-10-18T16:04:55.693 INFO:teuthology.orchestra.run.smithi088.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-10-18T16:04:55.693 INFO:teuthology.orchestra.run.smithi088.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-10-18T16:04:55.693 INFO:teuthology.orchestra.run.smithi088.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-10-18T16:04:55.693 INFO:teuthology.orchestra.run.smithi088.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-10-18T16:04:55.693 INFO:teuthology.orchestra.run.smithi088.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-10-18T16:04:55.694 INFO:teuthology.orchestra.run.smithi088.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-10-18T16:04:55.694 INFO:teuthology.orchestra.run.smithi088.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-10-18T16:04:55.694 INFO:teuthology.orchestra.run.smithi088.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-10-18T16:04:55.694 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-10-18T16:04:55.694 INFO:teuthology.orchestra.run.smithi088.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-10-18T16:04:55.695 INFO:teuthology.orchestra.run.smithi088.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-10-18T16:04:55.695 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-10-18T16:04:55.695 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-10-18T16:04:55.695 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-10-18T16:04:55.695 INFO:teuthology.orchestra.run.smithi088.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2023-10-18T16:04:55.695 INFO:teuthology.orchestra.run.smithi088.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-10-18T16:04:55.696 INFO:teuthology.orchestra.run.smithi088.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-10-18T16:04:55.696 INFO:teuthology.orchestra.run.smithi088.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-10-18T16:04:55.696 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-10-18T16:04:55.696 INFO:teuthology.orchestra.run.smithi088.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2023-10-18T16:04:55.696 INFO:teuthology.orchestra.run.smithi088.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-10-18T16:04:55.696 INFO:teuthology.orchestra.run.smithi088.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-10-18T16:04:55.697 INFO:teuthology.orchestra.run.smithi088.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-10-18T16:04:55.697 INFO:teuthology.orchestra.run.smithi088.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-10-18T16:04:55.697 INFO:teuthology.orchestra.run.smithi088.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-10-18T16:04:55.697 INFO:teuthology.orchestra.run.smithi088.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-10-18T16:04:55.697 INFO:teuthology.orchestra.run.smithi088.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-10-18T16:04:55.698 INFO:teuthology.orchestra.run.smithi088.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-10-18T16:04:55.698 INFO:teuthology.orchestra.run.smithi088.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-10-18T16:04:55.698 INFO:teuthology.orchestra.run.smithi088.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-10-18T16:04:55.698 INFO:teuthology.orchestra.run.smithi088.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2023-10-18T16:04:55.698 INFO:teuthology.orchestra.run.smithi088.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-10-18T16:04:55.698 INFO:teuthology.orchestra.run.smithi088.stdout: zlib-devel x86_64 1.2.11-17.el8 rhel-8-for-x86_64-baseos-rpms 58 k 2023-10-18T16:04:55.699 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:04:55.699 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2023-10-18T16:04:55.699 INFO:teuthology.orchestra.run.smithi088.stdout:====================================================================================================================== 2023-10-18T16:04:55.699 INFO:teuthology.orchestra.run.smithi088.stdout:Install 118 Packages 2023-10-18T16:04:55.699 INFO:teuthology.orchestra.run.smithi088.stdout:Upgrade 2 Packages 2023-10-18T16:04:55.699 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:04:55.700 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 201 M 2023-10-18T16:04:55.700 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2023-10-18T16:04:55.934 INFO:teuthology.orchestra.run.smithi088.stdout:(1/120): ceph-16.2.14-100.gb4ddd52c.el8.x86_64. 34 kB/s | 6.4 kB 00:00 2023-10-18T16:04:56.325 INFO:teuthology.orchestra.run.smithi088.stdout:(2/120): ceph-fuse-16.2.14-100.gb4ddd52c.el8.x8 2.3 MB/s | 912 kB 00:00 2023-10-18T16:04:56.373 INFO:teuthology.orchestra.run.smithi088.stdout:(3/120): ceph-immutable-object-cache-16.2.14-10 3.3 MB/s | 161 kB 00:00 2023-10-18T16:04:57.146 INFO:teuthology.orchestra.run.smithi088.stdout:(4/120): ceph-mds-16.2.14-100.gb4ddd52c.el8.x86 3.2 MB/s | 2.4 MB 00:00 2023-10-18T16:04:57.429 INFO:teuthology.orchestra.run.smithi088.stdout:(5/120): ceph-base-16.2.14-100.gb4ddd52c.el8.x8 3.6 MB/s | 6.1 MB 00:01 2023-10-18T16:04:57.507 INFO:teuthology.orchestra.run.smithi088.stdout:(6/120): ceph-mgr-16.2.14-100.gb4ddd52c.el8.x86 4.5 MB/s | 1.6 MB 00:00 2023-10-18T16:04:58.249 INFO:teuthology.orchestra.run.smithi088.stdout:(7/120): ceph-mon-16.2.14-100.gb4ddd52c.el8.x86 6.1 MB/s | 5.0 MB 00:00 2023-10-18T16:04:59.879 INFO:teuthology.orchestra.run.smithi088.stdout:(8/120): ceph-radosgw-16.2.14-100.gb4ddd52c.el8 7.1 MB/s | 12 MB 00:01 2023-10-18T16:04:59.927 INFO:teuthology.orchestra.run.smithi088.stdout:(9/120): ceph-selinux-16.2.14-100.gb4ddd52c.el8 541 kB/s | 26 kB 00:00 2023-10-18T16:05:00.271 INFO:teuthology.orchestra.run.smithi151.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-10-18T16:05:00.272 INFO:teuthology.orchestra.run.smithi151.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-10-18T16:05:00.283 INFO:teuthology.orchestra.run.smithi151.stdout:Package NetworkManager-1:1.30.0-13.el8_4.x86_64 is already installed. 2023-10-18T16:05:00.284 INFO:teuthology.orchestra.run.smithi151.stdout:Package iproute-5.9.0-4.el8.x86_64 is already installed. 2023-10-18T16:05:00.285 INFO:teuthology.orchestra.run.smithi151.stdout:Package util-linux-2.32.1-27.el8.x86_64 is already installed. 2023-10-18T16:05:00.286 INFO:teuthology.orchestra.run.smithi151.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-10-18T16:05:00.287 INFO:teuthology.orchestra.run.smithi151.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-10-18T16:05:00.290 INFO:teuthology.orchestra.run.smithi151.stdout:Package xfsdump-3.1.8-2.el8.x86_64 is already installed. 2023-10-18T16:05:00.291 INFO:teuthology.orchestra.run.smithi151.stdout:Package xfsprogs-5.0.0-8.el8.x86_64 is already installed. 2023-10-18T16:05:00.773 INFO:teuthology.orchestra.run.smithi088.stdout:(10/120): ceph-osd-16.2.14-100.gb4ddd52c.el8.x8 5.6 MB/s | 18 MB 00:03 2023-10-18T16:05:00.780 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2023-10-18T16:05:00.791 INFO:teuthology.orchestra.run.smithi151.stdout:====================================================================================================================== 2023-10-18T16:05:00.791 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2023-10-18T16:05:00.791 INFO:teuthology.orchestra.run.smithi151.stdout:====================================================================================================================== 2023-10-18T16:05:00.791 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2023-10-18T16:05:00.792 INFO:teuthology.orchestra.run.smithi151.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-10-18T16:05:00.792 INFO:teuthology.orchestra.run.smithi151.stdout: ceph x86_64 2:16.2.14-100.gb4ddd52c.el8 ceph 6.4 k 2023-10-18T16:05:00.792 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-base x86_64 2:16.2.14-100.gb4ddd52c.el8 ceph 6.1 M 2023-10-18T16:05:00.792 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-fuse x86_64 2:16.2.14-100.gb4ddd52c.el8 ceph 912 k 2023-10-18T16:05:00.792 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-immutable-object-cache x86_64 2:16.2.14-100.gb4ddd52c.el8 ceph 161 k 2023-10-18T16:05:00.793 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr x86_64 2:16.2.14-100.gb4ddd52c.el8 ceph 1.6 M 2023-10-18T16:05:00.793 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-cephadm noarch 2:16.2.14-100.gb4ddd52c.el8 ceph-noarch 116 k 2023-10-18T16:05:00.793 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-dashboard noarch 2:16.2.14-100.gb4ddd52c.el8 ceph-noarch 3.1 M 2023-10-18T16:05:00.793 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.14-100.gb4ddd52c.el8 ceph-noarch 8.6 M 2023-10-18T16:05:00.794 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-rook noarch 2:16.2.14-100.gb4ddd52c.el8 ceph-noarch 27 k 2023-10-18T16:05:00.794 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-radosgw x86_64 2:16.2.14-100.gb4ddd52c.el8 ceph 12 M 2023-10-18T16:05:00.794 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-test x86_64 2:16.2.14-100.gb4ddd52c.el8 ceph 52 M 2023-10-18T16:05:00.794 INFO:teuthology.orchestra.run.smithi151.stdout: cephadm noarch 2:16.2.14-100.gb4ddd52c.el8 ceph-noarch 84 k 2023-10-18T16:05:00.794 INFO:teuthology.orchestra.run.smithi151.stdout: cephfs-mirror x86_64 2:16.2.14-100.gb4ddd52c.el8 ceph 257 k 2023-10-18T16:05:00.795 INFO:teuthology.orchestra.run.smithi151.stdout: cephfs-top noarch 2:16.2.14-100.gb4ddd52c.el8 ceph-noarch 19 k 2023-10-18T16:05:00.795 INFO:teuthology.orchestra.run.smithi151.stdout: elfutils-libelf-devel x86_64 0.182-3.el8 rhel-8-for-x86_64-baseos-rpms 59 k 2023-10-18T16:05:00.795 INFO:teuthology.orchestra.run.smithi151.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-10-18T16:05:00.795 INFO:teuthology.orchestra.run.smithi151.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-10-18T16:05:00.796 INFO:teuthology.orchestra.run.smithi151.stdout: libcephfs-devel x86_64 2:16.2.14-100.gb4ddd52c.el8 ceph 25 k 2023-10-18T16:05:00.796 INFO:teuthology.orchestra.run.smithi151.stdout: libcephfs2 x86_64 2:16.2.14-100.gb4ddd52c.el8 ceph 825 k 2023-10-18T16:05:00.796 INFO:teuthology.orchestra.run.smithi151.stdout: librados-devel x86_64 2:16.2.14-100.gb4ddd52c.el8 ceph 131 k 2023-10-18T16:05:00.796 INFO:teuthology.orchestra.run.smithi151.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-10-18T16:05:00.796 INFO:teuthology.orchestra.run.smithi151.stdout: libuuid-devel x86_64 2.32.1-27.el8 rhel-8-for-x86_64-baseos-rpms 97 k 2023-10-18T16:05:00.797 INFO:teuthology.orchestra.run.smithi151.stdout: openssl-devel x86_64 1:1.1.1g-15.el8_3 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-10-18T16:05:00.797 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cephfs x86_64 2:16.2.14-100.gb4ddd52c.el8 ceph 215 k 2023-10-18T16:05:00.797 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rados x86_64 2:16.2.14-100.gb4ddd52c.el8 ceph 390 k 2023-10-18T16:05:00.797 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rbd x86_64 2:16.2.14-100.gb4ddd52c.el8 ceph 369 k 2023-10-18T16:05:00.798 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rgw x86_64 2:16.2.14-100.gb4ddd52c.el8 ceph 114 k 2023-10-18T16:05:00.798 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-fuse x86_64 2:16.2.14-100.gb4ddd52c.el8 ceph 88 k 2023-10-18T16:05:00.798 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-mirror x86_64 2:16.2.14-100.gb4ddd52c.el8 ceph 3.7 M 2023-10-18T16:05:00.798 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-nbd x86_64 2:16.2.14-100.gb4ddd52c.el8 ceph 173 k 2023-10-18T16:05:00.799 INFO:teuthology.orchestra.run.smithi151.stdout: xfsprogs-devel x86_64 5.0.0-8.el8 rhel-8-for-x86_64-baseos-rpms 70 k 2023-10-18T16:05:00.799 INFO:teuthology.orchestra.run.smithi151.stdout:Upgrading: 2023-10-18T16:05:00.799 INFO:teuthology.orchestra.run.smithi151.stdout: librados2 x86_64 2:16.2.14-100.gb4ddd52c.el8 ceph 3.7 M 2023-10-18T16:05:00.799 INFO:teuthology.orchestra.run.smithi151.stdout: librbd1 x86_64 2:16.2.14-100.gb4ddd52c.el8 ceph 4.2 M 2023-10-18T16:05:00.800 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2023-10-18T16:05:00.800 INFO:teuthology.orchestra.run.smithi151.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-10-18T16:05:00.800 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-common x86_64 2:16.2.14-100.gb4ddd52c.el8 ceph 24 M 2023-10-18T16:05:00.800 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-grafana-dashboards noarch 2:16.2.14-100.gb4ddd52c.el8 ceph-noarch 24 k 2023-10-18T16:05:00.800 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mds x86_64 2:16.2.14-100.gb4ddd52c.el8 ceph 2.4 M 2023-10-18T16:05:00.801 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-modules-core noarch 2:16.2.14-100.gb4ddd52c.el8 ceph-noarch 244 k 2023-10-18T16:05:00.801 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mon x86_64 2:16.2.14-100.gb4ddd52c.el8 ceph 5.0 M 2023-10-18T16:05:00.801 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-osd x86_64 2:16.2.14-100.gb4ddd52c.el8 ceph 18 M 2023-10-18T16:05:00.801 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-prometheus-alerts noarch 2:16.2.14-100.gb4ddd52c.el8 ceph-noarch 15 k 2023-10-18T16:05:00.801 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-selinux x86_64 2:16.2.14-100.gb4ddd52c.el8 ceph 26 k 2023-10-18T16:05:00.802 INFO:teuthology.orchestra.run.smithi151.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-10-18T16:05:00.802 INFO:teuthology.orchestra.run.smithi151.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-10-18T16:05:00.802 INFO:teuthology.orchestra.run.smithi151.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2023-10-18T16:05:00.802 INFO:teuthology.orchestra.run.smithi151.stdout: keyutils-libs-devel x86_64 1.5.10-6.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-10-18T16:05:00.803 INFO:teuthology.orchestra.run.smithi151.stdout: krb5-devel x86_64 1.18.2-8.3.el8_4 rhel-8-for-x86_64-baseos-rpms 559 k 2023-10-18T16:05:00.803 INFO:teuthology.orchestra.run.smithi151.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2023-10-18T16:05:00.803 INFO:teuthology.orchestra.run.smithi151.stdout: libcephsqlite x86_64 2:16.2.14-100.gb4ddd52c.el8 ceph 174 k 2023-10-18T16:05:00.803 INFO:teuthology.orchestra.run.smithi151.stdout: libcom_err-devel x86_64 1.45.6-1.el8 rhel-8-for-x86_64-baseos-rpms 38 k 2023-10-18T16:05:00.803 INFO:teuthology.orchestra.run.smithi151.stdout: libgfortran x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2023-10-18T16:05:00.804 INFO:teuthology.orchestra.run.smithi151.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-10-18T16:05:00.804 INFO:teuthology.orchestra.run.smithi151.stdout: libquadmath x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2023-10-18T16:05:00.804 INFO:teuthology.orchestra.run.smithi151.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-10-18T16:05:00.804 INFO:teuthology.orchestra.run.smithi151.stdout: libradosstriper1 x86_64 2:16.2.14-100.gb4ddd52c.el8 ceph 494 k 2023-10-18T16:05:00.804 INFO:teuthology.orchestra.run.smithi151.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2023-10-18T16:05:00.805 INFO:teuthology.orchestra.run.smithi151.stdout: librgw2 x86_64 2:16.2.14-100.gb4ddd52c.el8 ceph 3.8 M 2023-10-18T16:05:00.805 INFO:teuthology.orchestra.run.smithi151.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-10-18T16:05:00.805 INFO:teuthology.orchestra.run.smithi151.stdout: libsepol-devel x86_64 2.9-2.el8 rhel-8-for-x86_64-baseos-rpms 86 k 2023-10-18T16:05:00.805 INFO:teuthology.orchestra.run.smithi151.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-10-18T16:05:00.805 INFO:teuthology.orchestra.run.smithi151.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-10-18T16:05:00.806 INFO:teuthology.orchestra.run.smithi151.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-10-18T16:05:00.806 INFO:teuthology.orchestra.run.smithi151.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-10-18T16:05:00.806 INFO:teuthology.orchestra.run.smithi151.stdout: openblas x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.6 M 2023-10-18T16:05:00.806 INFO:teuthology.orchestra.run.smithi151.stdout: openblas-threads x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-10-18T16:05:00.807 INFO:teuthology.orchestra.run.smithi151.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-10-18T16:05:00.807 INFO:teuthology.orchestra.run.smithi151.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-10-18T16:05:00.807 INFO:teuthology.orchestra.run.smithi151.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-10-18T16:05:00.807 INFO:teuthology.orchestra.run.smithi151.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2023-10-18T16:05:00.807 INFO:teuthology.orchestra.run.smithi151.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-10-18T16:05:00.808 INFO:teuthology.orchestra.run.smithi151.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-10-18T16:05:00.808 INFO:teuthology.orchestra.run.smithi151.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-10-18T16:05:00.808 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-10-18T16:05:00.808 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ceph-argparse x86_64 2:16.2.14-100.gb4ddd52c.el8 ceph 45 k 2023-10-18T16:05:00.809 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ceph-common x86_64 2:16.2.14-100.gb4ddd52c.el8 ceph 99 k 2023-10-18T16:05:00.809 INFO:teuthology.orchestra.run.smithi151.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-10-18T16:05:00.809 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-10-18T16:05:00.809 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-10-18T16:05:00.810 INFO:teuthology.orchestra.run.smithi151.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2023-10-18T16:05:00.810 INFO:teuthology.orchestra.run.smithi151.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-10-18T16:05:00.810 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-10-18T16:05:00.810 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-10-18T16:05:00.811 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2023-10-18T16:05:00.811 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-10-18T16:05:00.811 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-10-18T16:05:00.811 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-10-18T16:05:00.811 INFO:teuthology.orchestra.run.smithi151.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-10-18T16:05:00.812 INFO:teuthology.orchestra.run.smithi151.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-10-18T16:05:00.812 INFO:teuthology.orchestra.run.smithi151.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-10-18T16:05:00.812 INFO:teuthology.orchestra.run.smithi151.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-10-18T16:05:00.812 INFO:teuthology.orchestra.run.smithi151.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-10-18T16:05:00.812 INFO:teuthology.orchestra.run.smithi151.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-10-18T16:05:00.813 INFO:teuthology.orchestra.run.smithi151.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-10-18T16:05:00.813 INFO:teuthology.orchestra.run.smithi151.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-10-18T16:05:00.813 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-10-18T16:05:00.813 INFO:teuthology.orchestra.run.smithi151.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-10-18T16:05:00.813 INFO:teuthology.orchestra.run.smithi151.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-10-18T16:05:00.814 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-10-18T16:05:00.814 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-10-18T16:05:00.814 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-10-18T16:05:00.814 INFO:teuthology.orchestra.run.smithi151.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2023-10-18T16:05:00.815 INFO:teuthology.orchestra.run.smithi151.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-10-18T16:05:00.815 INFO:teuthology.orchestra.run.smithi151.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-10-18T16:05:00.815 INFO:teuthology.orchestra.run.smithi151.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-10-18T16:05:00.815 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-10-18T16:05:00.815 INFO:teuthology.orchestra.run.smithi151.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2023-10-18T16:05:00.816 INFO:teuthology.orchestra.run.smithi151.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-10-18T16:05:00.816 INFO:teuthology.orchestra.run.smithi151.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-10-18T16:05:00.816 INFO:teuthology.orchestra.run.smithi151.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-10-18T16:05:00.816 INFO:teuthology.orchestra.run.smithi151.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-10-18T16:05:00.816 INFO:teuthology.orchestra.run.smithi151.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-10-18T16:05:00.817 INFO:teuthology.orchestra.run.smithi151.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-10-18T16:05:00.817 INFO:teuthology.orchestra.run.smithi151.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-10-18T16:05:00.817 INFO:teuthology.orchestra.run.smithi151.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-10-18T16:05:00.817 INFO:teuthology.orchestra.run.smithi151.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-10-18T16:05:00.817 INFO:teuthology.orchestra.run.smithi151.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-10-18T16:05:00.818 INFO:teuthology.orchestra.run.smithi151.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2023-10-18T16:05:00.818 INFO:teuthology.orchestra.run.smithi151.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-10-18T16:05:00.818 INFO:teuthology.orchestra.run.smithi151.stdout: zlib-devel x86_64 1.2.11-17.el8 rhel-8-for-x86_64-baseos-rpms 58 k 2023-10-18T16:05:00.818 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-10-18T16:05:00.818 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2023-10-18T16:05:00.819 INFO:teuthology.orchestra.run.smithi151.stdout:====================================================================================================================== 2023-10-18T16:05:00.819 INFO:teuthology.orchestra.run.smithi151.stdout:Install 118 Packages 2023-10-18T16:05:00.819 INFO:teuthology.orchestra.run.smithi151.stdout:Upgrade 2 Packages 2023-10-18T16:05:00.819 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-10-18T16:05:00.820 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 201 M 2023-10-18T16:05:00.820 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2023-10-18T16:05:00.826 INFO:teuthology.orchestra.run.smithi088.stdout:(11/120): cephfs-mirror-16.2.14-100.gb4ddd52c.e 4.8 MB/s | 257 kB 00:00 2023-10-18T16:05:00.863 INFO:teuthology.orchestra.run.smithi088.stdout:(12/120): libcephfs-devel-16.2.14-100.gb4ddd52c 692 kB/s | 25 kB 00:00 2023-10-18T16:05:00.974 INFO:teuthology.orchestra.run.smithi088.stdout:(13/120): libcephfs2-16.2.14-100.gb4ddd52c.el8. 7.3 MB/s | 825 kB 00:00 2023-10-18T16:05:01.013 INFO:teuthology.orchestra.run.smithi088.stdout:(14/120): libcephsqlite-16.2.14-100.gb4ddd52c.e 4.4 MB/s | 174 kB 00:00 2023-10-18T16:05:01.051 INFO:teuthology.orchestra.run.smithi088.stdout:(15/120): librados-devel-16.2.14-100.gb4ddd52c. 3.4 MB/s | 131 kB 00:00 2023-10-18T16:05:01.072 INFO:teuthology.orchestra.run.smithi151.stdout:(1/120): ceph-16.2.14-100.gb4ddd52c.el8.x86_64. 64 kB/s | 6.4 kB 00:00 2023-10-18T16:05:01.125 INFO:teuthology.orchestra.run.smithi088.stdout:(16/120): libradosstriper1-16.2.14-100.gb4ddd52 6.5 MB/s | 494 kB 00:00 2023-10-18T16:05:01.339 INFO:teuthology.orchestra.run.smithi151.stdout:(2/120): ceph-fuse-16.2.14-100.gb4ddd52c.el8.x8 3.3 MB/s | 912 kB 00:00 2023-10-18T16:05:01.377 INFO:teuthology.orchestra.run.smithi151.stdout:(3/120): ceph-immutable-object-cache-16.2.14-10 4.2 MB/s | 161 kB 00:00 2023-10-18T16:05:01.580 INFO:teuthology.orchestra.run.smithi088.stdout:(17/120): ceph-common-16.2.14-100.gb4ddd52c.el8 4.1 MB/s | 24 MB 00:05 2023-10-18T16:05:01.634 INFO:teuthology.orchestra.run.smithi088.stdout:(18/120): python3-ceph-argparse-16.2.14-100.gb4 833 kB/s | 45 kB 00:00 2023-10-18T16:05:01.690 INFO:teuthology.orchestra.run.smithi088.stdout:(19/120): librgw2-16.2.14-100.gb4ddd52c.el8.x86 6.7 MB/s | 3.8 MB 00:00 2023-10-18T16:05:01.691 INFO:teuthology.orchestra.run.smithi088.stdout:(20/120): python3-ceph-common-16.2.14-100.gb4dd 1.7 MB/s | 99 kB 00:00 2023-10-18T16:05:01.741 INFO:teuthology.orchestra.run.smithi088.stdout:(21/120): python3-cephfs-16.2.14-100.gb4ddd52c. 4.2 MB/s | 215 kB 00:00 2023-10-18T16:05:01.758 INFO:teuthology.orchestra.run.smithi151.stdout:(4/120): ceph-mds-16.2.14-100.gb4ddd52c.el8.x86 6.4 MB/s | 2.4 MB 00:00 2023-10-18T16:05:01.799 INFO:teuthology.orchestra.run.smithi088.stdout:(22/120): python3-rbd-16.2.14-100.gb4ddd52c.el8 6.2 MB/s | 369 kB 00:00 2023-10-18T16:05:01.801 INFO:teuthology.orchestra.run.smithi088.stdout:(23/120): python3-rados-16.2.14-100.gb4ddd52c.e 3.5 MB/s | 390 kB 00:00 2023-10-18T16:05:01.852 INFO:teuthology.orchestra.run.smithi088.stdout:(24/120): python3-rgw-16.2.14-100.gb4ddd52c.el8 2.1 MB/s | 114 kB 00:00 2023-10-18T16:05:01.859 INFO:teuthology.orchestra.run.smithi088.stdout:(25/120): rbd-fuse-16.2.14-100.gb4ddd52c.el8.x8 1.5 MB/s | 88 kB 00:00 2023-10-18T16:05:01.918 INFO:teuthology.orchestra.run.smithi088.stdout:(26/120): rbd-nbd-16.2.14-100.gb4ddd52c.el8.x86 2.9 MB/s | 173 kB 00:00 2023-10-18T16:05:01.979 INFO:teuthology.orchestra.run.smithi088.stdout:(27/120): ceph-grafana-dashboards-16.2.14-100.g 404 kB/s | 24 kB 00:00 2023-10-18T16:05:01.988 INFO:teuthology.orchestra.run.smithi151.stdout:(5/120): ceph-mgr-16.2.14-100.gb4ddd52c.el8.x86 7.0 MB/s | 1.6 MB 00:00 2023-10-18T16:05:02.032 INFO:teuthology.orchestra.run.smithi088.stdout:(28/120): ceph-mgr-cephadm-16.2.14-100.gb4ddd52 2.1 MB/s | 116 kB 00:00 2023-10-18T16:05:02.535 INFO:teuthology.orchestra.run.smithi088.stdout:(29/120): rbd-mirror-16.2.14-100.gb4ddd52c.el8. 5.4 MB/s | 3.7 MB 00:00 2023-10-18T16:05:02.685 INFO:teuthology.orchestra.run.smithi151.stdout:(6/120): ceph-mon-16.2.14-100.gb4ddd52c.el8.x86 7.2 MB/s | 5.0 MB 00:00 2023-10-18T16:05:02.900 INFO:teuthology.orchestra.run.smithi151.stdout:(7/120): ceph-base-16.2.14-100.gb4ddd52c.el8.x8 3.1 MB/s | 6.1 MB 00:01 2023-10-18T16:05:02.989 INFO:teuthology.orchestra.run.smithi088.stdout:(30/120): ceph-mgr-dashboard-16.2.14-100.gb4ddd 3.3 MB/s | 3.1 MB 00:00 2023-10-18T16:05:03.070 INFO:teuthology.orchestra.run.smithi088.stdout:(31/120): ceph-mgr-modules-core-16.2.14-100.gb4 2.9 MB/s | 244 kB 00:00 2023-10-18T16:05:03.151 INFO:teuthology.orchestra.run.smithi088.stdout:(32/120): ceph-mgr-rook-16.2.14-100.gb4ddd52c.e 342 kB/s | 27 kB 00:00 2023-10-18T16:05:03.241 INFO:teuthology.orchestra.run.smithi088.stdout:(33/120): ceph-prometheus-alerts-16.2.14-100.gb 165 kB/s | 15 kB 00:00 2023-10-18T16:05:03.321 INFO:teuthology.orchestra.run.smithi088.stdout:(34/120): cephadm-16.2.14-100.gb4ddd52c.el8.noa 1.0 MB/s | 84 kB 00:00 2023-10-18T16:05:03.401 INFO:teuthology.orchestra.run.smithi088.stdout:(35/120): cephfs-top-16.2.14-100.gb4ddd52c.el8. 234 kB/s | 19 kB 00:00 2023-10-18T16:05:03.566 INFO:teuthology.orchestra.run.smithi088.stdout:(36/120): gperftools-libs-2.7-9.el8.x86_64.rpm 1.8 MB/s | 306 kB 00:00 2023-10-18T16:05:03.591 INFO:teuthology.orchestra.run.smithi088.stdout:(37/120): leveldb-1.22-1.el8.x86_64.rpm 7.0 MB/s | 181 kB 00:00 2023-10-18T16:05:03.615 INFO:teuthology.orchestra.run.smithi088.stdout:(38/120): liboath-2.6.2-3.el8.x86_64.rpm 2.4 MB/s | 59 kB 00:00 2023-10-18T16:05:03.642 INFO:teuthology.orchestra.run.smithi088.stdout:(39/120): libunwind-1.3.1-3.el8.x86_64.rpm 2.7 MB/s | 75 kB 00:00 2023-10-18T16:05:03.668 INFO:teuthology.orchestra.run.smithi088.stdout:(40/120): python3-apipkg-1.5-6.el8.noarch.rpm 816 kB/s | 21 kB 00:00 2023-10-18T16:05:03.691 INFO:teuthology.orchestra.run.smithi088.stdout:(41/120): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 1.8 MB/s | 44 kB 00:00 2023-10-18T16:05:03.716 INFO:teuthology.orchestra.run.smithi088.stdout:(42/120): python3-beautifulsoup4-4.6.3-2.el8.1. 7.4 MB/s | 185 kB 00:00 2023-10-18T16:05:03.739 INFO:teuthology.orchestra.run.smithi088.stdout:(43/120): python3-cachetools-3.1.1-4.el8.noarch 1.4 MB/s | 33 kB 00:00 2023-10-18T16:05:03.769 INFO:teuthology.orchestra.run.smithi088.stdout:(44/120): python3-certifi-2018.10.15-7.el8.noar 554 kB/s | 16 kB 00:00 2023-10-18T16:05:03.803 INFO:teuthology.orchestra.run.smithi088.stdout:(45/120): python3-cheroot-8.5.2-1.el8.noarch.rp 4.9 MB/s | 173 kB 00:00 2023-10-18T16:05:03.849 INFO:teuthology.orchestra.run.smithi088.stdout:(46/120): python3-cherrypy-18.4.0-1.el8.noarch. 8.3 MB/s | 384 kB 00:00 2023-10-18T16:05:03.876 INFO:teuthology.orchestra.run.smithi088.stdout:(47/120): python3-execnet-1.7.1-1.el8.noarch.rp 13 MB/s | 355 kB 00:00 2023-10-18T16:05:03.899 INFO:teuthology.orchestra.run.smithi088.stdout:(48/120): python3-google-auth-1.1.1-10.el8.noar 3.4 MB/s | 81 kB 00:00 2023-10-18T16:05:03.922 INFO:teuthology.orchestra.run.smithi088.stdout:(49/120): python3-jaraco-6.2-6.el8.noarch.rpm 487 kB/s | 11 kB 00:00 2023-10-18T16:05:03.945 INFO:teuthology.orchestra.run.smithi088.stdout:(50/120): python3-jaraco-functools-2.0-4.el8.no 769 kB/s | 18 kB 00:00 2023-10-18T16:05:04.020 INFO:teuthology.orchestra.run.smithi088.stdout:(51/120): python3-kubernetes-11.0.0-6.el8.noarc 16 MB/s | 1.2 MB 00:00 2023-10-18T16:05:04.043 INFO:teuthology.orchestra.run.smithi088.stdout:(52/120): python3-logutils-0.3.5-11.el8.noarch. 2.1 MB/s | 49 kB 00:00 2023-10-18T16:05:04.067 INFO:teuthology.orchestra.run.smithi088.stdout:(53/120): python3-more-itertools-7.2.0-3.el8.no 2.5 MB/s | 59 kB 00:00 2023-10-18T16:05:04.093 INFO:teuthology.orchestra.run.smithi088.stdout:(54/120): python3-pecan-1.3.2-9.el8.noarch.rpm 11 MB/s | 283 kB 00:00 2023-10-18T16:05:04.115 INFO:teuthology.orchestra.run.smithi088.stdout:(55/120): python3-portend-2.6-1.el8.noarch.rpm 687 kB/s | 16 kB 00:00 2023-10-18T16:05:04.139 INFO:teuthology.orchestra.run.smithi088.stdout:(56/120): python3-remoto-1.2.1-1.el8.noarch.rpm 2.1 MB/s | 49 kB 00:00 2023-10-18T16:05:04.161 INFO:teuthology.orchestra.run.smithi088.stdout:(57/120): python3-repoze-lru-0.7-6.el8.noarch.r 1.4 MB/s | 34 kB 00:00 2023-10-18T16:05:04.186 INFO:teuthology.orchestra.run.smithi088.stdout:(58/120): python3-routes-2.4.1-12.el8.noarch.rp 7.8 MB/s | 196 kB 00:00 2023-10-18T16:05:04.210 INFO:teuthology.orchestra.run.smithi088.stdout:(59/120): python3-rsa-4.9-2.el8.noarch.rpm 2.7 MB/s | 64 kB 00:00 2023-10-18T16:05:04.232 INFO:teuthology.orchestra.run.smithi088.stdout:(60/120): python3-simplegeneric-0.8.1-17.el8.no 829 kB/s | 19 kB 00:00 2023-10-18T16:05:04.255 INFO:teuthology.orchestra.run.smithi088.stdout:(61/120): python3-singledispatch-3.4.0.3-18.el8 1.0 MB/s | 24 kB 00:00 2023-10-18T16:05:04.278 INFO:teuthology.orchestra.run.smithi088.stdout:(62/120): python3-tempora-1.14.1-5.el8.noarch.r 1.3 MB/s | 29 kB 00:00 2023-10-18T16:05:04.301 INFO:teuthology.orchestra.run.smithi088.stdout:(63/120): python3-trustme-0.6.0-4.el8.noarch.rp 1.2 MB/s | 27 kB 00:00 2023-10-18T16:05:04.327 INFO:teuthology.orchestra.run.smithi088.stdout:(64/120): python3-waitress-1.4.3-1.el8.noarch.r 9.8 MB/s | 255 kB 00:00 2023-10-18T16:05:04.352 INFO:teuthology.orchestra.run.smithi088.stdout:(65/120): python3-webob-1.8.5-1.el8.1.noarch.rp 9.7 MB/s | 251 kB 00:00 2023-10-18T16:05:04.376 INFO:teuthology.orchestra.run.smithi088.stdout:(66/120): python3-websocket-client-0.56.0-5.el8 2.6 MB/s | 61 kB 00:00 2023-10-18T16:05:04.399 INFO:teuthology.orchestra.run.smithi088.stdout:(67/120): python3-webtest-2.0.33-1.el8.noarch.r 3.6 MB/s | 85 kB 00:00 2023-10-18T16:05:04.422 INFO:teuthology.orchestra.run.smithi088.stdout:(68/120): python3-zc-lockfile-2.0-2.el8.noarch. 1.0 MB/s | 23 kB 00:00 2023-10-18T16:05:04.427 INFO:teuthology.orchestra.run.smithi088.stdout:(69/120): xmlstarlet-1.6.1-20.el8.x86_64.rpm 16 MB/s | 70 kB 00:00 2023-10-18T16:05:04.473 INFO:teuthology.orchestra.run.smithi088.stdout:(70/120): ceph-mgr-diskprediction-local-16.2.14 4.4 MB/s | 8.6 MB 00:01 2023-10-18T16:05:04.474 INFO:teuthology.orchestra.run.smithi088.stdout:(71/120): libaio-devel-0.3.112-1.el8.x86_64.rpm 397 kB/s | 19 kB 00:00 2023-10-18T16:05:04.489 INFO:teuthology.orchestra.run.smithi088.stdout:(72/120): python3-oauthlib-2.1.0-1.el8.noarch.r 9.9 MB/s | 155 kB 00:00 2023-10-18T16:05:04.504 INFO:teuthology.orchestra.run.smithi088.stdout:(73/120): python3-jwt-1.6.1-2.el8.noarch.rpm 1.3 MB/s | 43 kB 00:00 2023-10-18T16:05:04.507 INFO:teuthology.orchestra.run.smithi088.stdout:(74/120): pcre2-utf32-10.32-2.el8.x86_64.rpm 12 MB/s | 220 kB 00:00 2023-10-18T16:05:04.520 INFO:teuthology.orchestra.run.smithi088.stdout:(75/120): keyutils-libs-devel-1.5.10-6.el8.x86_ 3.0 MB/s | 48 kB 00:00 2023-10-18T16:05:04.525 INFO:teuthology.orchestra.run.smithi088.stdout:(76/120): pcre2-utf16-10.32-2.el8.x86_64.rpm 13 MB/s | 229 kB 00:00 2023-10-18T16:05:04.532 INFO:teuthology.orchestra.run.smithi151.stdout:(8/120): ceph-radosgw-16.2.14-100.gb4ddd52c.el8 7.1 MB/s | 12 MB 00:01 2023-10-18T16:05:04.537 INFO:teuthology.orchestra.run.smithi088.stdout:(77/120): python3-requests-oauthlib-1.0.0-1.el8 2.4 MB/s | 43 kB 00:00 2023-10-18T16:05:04.552 INFO:teuthology.orchestra.run.smithi088.stdout:(78/120): libverto-devel-0.3.0-5.el8.x86_64.rpm 1.3 MB/s | 18 kB 00:00 2023-10-18T16:05:04.563 INFO:teuthology.orchestra.run.smithi088.stdout:(79/120): openssl-devel-1.1.1g-15.el8_3.x86_64. 61 MB/s | 2.3 MB 00:00 2023-10-18T16:05:04.571 INFO:teuthology.orchestra.run.smithi088.stdout:(80/120): pcre2-devel-10.32-2.el8.x86_64.rpm 31 MB/s | 605 kB 00:00 2023-10-18T16:05:04.578 INFO:teuthology.orchestra.run.smithi088.stdout:(81/120): fuse-2.9.7-12.el8.x86_64.rpm 5.3 MB/s | 83 kB 00:00 2023-10-18T16:05:04.584 INFO:teuthology.orchestra.run.smithi151.stdout:(9/120): ceph-selinux-16.2.14-100.gb4ddd52c.el8 499 kB/s | 26 kB 00:00 2023-10-18T16:05:04.588 INFO:teuthology.orchestra.run.smithi088.stdout:(82/120): elfutils-libelf-devel-0.182-3.el8.x86 3.3 MB/s | 59 kB 00:00 2023-10-18T16:05:04.592 INFO:teuthology.orchestra.run.smithi088.stdout:(83/120): libquadmath-8.4.1-1.el8.x86_64.rpm 12 MB/s | 169 kB 00:00 2023-10-18T16:05:04.605 INFO:teuthology.orchestra.run.smithi088.stdout:(84/120): libsepol-devel-2.9-2.el8.x86_64.rpm 6.6 MB/s | 86 kB 00:00 2023-10-18T16:05:04.606 INFO:teuthology.orchestra.run.smithi088.stdout:(85/120): libcom_err-devel-1.45.6-1.el8.x86_64. 2.2 MB/s | 38 kB 00:00 2023-10-18T16:05:04.620 INFO:teuthology.orchestra.run.smithi088.stdout:(86/120): zlib-devel-1.2.11-17.el8.x86_64.rpm 4.0 MB/s | 58 kB 00:00 2023-10-18T16:05:04.623 INFO:teuthology.orchestra.run.smithi088.stdout:(87/120): librabbitmq-0.9.0-3.el8.x86_64.rpm 2.8 MB/s | 47 kB 00:00 2023-10-18T16:05:04.644 INFO:teuthology.orchestra.run.smithi088.stdout:(88/120): krb5-devel-1.18.2-8.3.el8_4.x86_64.rp 23 MB/s | 559 kB 00:00 2023-10-18T16:05:04.646 INFO:teuthology.orchestra.run.smithi088.stdout:(89/120): libgfortran-8.4.1-1.el8.x86_64.rpm 27 MB/s | 639 kB 00:00 2023-10-18T16:05:04.660 INFO:teuthology.orchestra.run.smithi088.stdout:(90/120): xfsprogs-devel-5.0.0-8.el8.x86_64.rpm 4.2 MB/s | 70 kB 00:00 2023-10-18T16:05:04.664 INFO:teuthology.orchestra.run.smithi088.stdout:(91/120): libselinux-devel-2.9-5.el8.x86_64.rpm 11 MB/s | 200 kB 00:00 2023-10-18T16:05:04.676 INFO:teuthology.orchestra.run.smithi088.stdout:(92/120): libuuid-devel-2.32.1-27.el8.x86_64.rp 5.7 MB/s | 97 kB 00:00 2023-10-18T16:05:04.693 INFO:teuthology.orchestra.run.smithi088.stdout:(93/120): python3-prettytable-0.7.2-14.el8.noar 2.6 MB/s | 44 kB 00:00 2023-10-18T16:05:04.710 INFO:teuthology.orchestra.run.smithi088.stdout:(94/120): python3-jsonpatch-1.21-2.el8.noarch.r 1.6 MB/s | 27 kB 00:00 2023-10-18T16:05:04.727 INFO:teuthology.orchestra.run.smithi088.stdout:(95/120): jq-1.5-12.el8.x86_64.rpm 9.2 MB/s | 161 kB 00:00 2023-10-18T16:05:04.746 INFO:teuthology.orchestra.run.smithi088.stdout:(96/120): flex-2.6.1-9.el8.x86_64.rpm 17 MB/s | 320 kB 00:00 2023-10-18T16:05:04.766 INFO:teuthology.orchestra.run.smithi088.stdout:(97/120): python3-werkzeug-0.12.2-4.el8.noarch. 23 MB/s | 457 kB 00:00 2023-10-18T16:05:04.922 INFO:teuthology.orchestra.run.smithi088.stdout:(98/120): python3-scipy-1.0.0-20.module+el8.1.0 56 MB/s | 15 MB 00:00 2023-10-18T16:05:04.937 INFO:teuthology.orchestra.run.smithi088.stdout:(99/120): python3-markupsafe-0.23-19.el8.x86_64 2.6 MB/s | 39 kB 00:00 2023-10-18T16:05:04.950 INFO:teuthology.orchestra.run.smithi088.stdout:(100/120): python3-babel-2.5.1-5.el8.noarch.rpm 26 MB/s | 4.8 MB 00:00 2023-10-18T16:05:04.961 INFO:teuthology.orchestra.run.smithi088.stdout:(101/120): bison-3.0.4-10.el8.x86_64.rpm 29 MB/s | 688 kB 00:00 2023-10-18T16:05:04.967 INFO:teuthology.orchestra.run.smithi088.stdout:(102/120): lttng-ust-2.8.1-11.el8.x86_64.rpm 15 MB/s | 259 kB 00:00 2023-10-18T16:05:04.980 INFO:teuthology.orchestra.run.smithi088.stdout:(103/120): python3-numpy-f2py-1.14.3-9.el8.x86_ 12 MB/s | 225 kB 00:00 2023-10-18T16:05:04.981 INFO:teuthology.orchestra.run.smithi088.stdout:(104/120): python3-pyasn1-0.3.7-6.el8.noarch.rp 9.0 MB/s | 126 kB 00:00 2023-10-18T16:05:05.000 INFO:teuthology.orchestra.run.smithi088.stdout:(105/120): oniguruma-6.8.2-2.el8.x86_64.rpm 9.8 MB/s | 187 kB 00:00 2023-10-18T16:05:05.022 INFO:teuthology.orchestra.run.smithi088.stdout:(106/120): python3-jinja2-2.10.1-2.el8_0.noarch 24 MB/s | 538 kB 00:00 2023-10-18T16:05:05.037 INFO:teuthology.orchestra.run.smithi088.stdout:(107/120): python3-numpy-1.14.3-9.el8.x86_64.rp 66 MB/s | 3.7 MB 00:00 2023-10-18T16:05:05.041 INFO:teuthology.orchestra.run.smithi088.stdout:(108/120): socat-1.7.3.3-2.el8.x86_64.rpm 16 MB/s | 302 kB 00:00 2023-10-18T16:05:05.051 INFO:teuthology.orchestra.run.smithi088.stdout:(109/120): python3-mako-1.0.6-13.el8.noarch.rpm 11 MB/s | 157 kB 00:00 2023-10-18T16:05:05.058 INFO:teuthology.orchestra.run.smithi088.stdout:(110/120): python3-pyasn1-modules-0.3.7-6.el8.n 6.1 MB/s | 110 kB 00:00 2023-10-18T16:05:05.069 INFO:teuthology.orchestra.run.smithi088.stdout:(111/120): librdkafka-0.11.4-1.el8.x86_64.rpm 20 MB/s | 353 kB 00:00 2023-10-18T16:05:05.074 INFO:teuthology.orchestra.run.smithi088.stdout:(112/120): python3-jsonpointer-1.10-11.el8.noar 1.3 MB/s | 20 kB 00:00 2023-10-18T16:05:05.090 INFO:teuthology.orchestra.run.smithi088.stdout:(113/120): libtool-2.4.6-25.el8.x86_64.rpm 33 MB/s | 709 kB 00:00 2023-10-18T16:05:05.091 INFO:teuthology.orchestra.run.smithi088.stdout:(114/120): python3-pytz-2017.2-9.el8.noarch.rpm 3.2 MB/s | 54 kB 00:00 2023-10-18T16:05:05.169 INFO:teuthology.orchestra.run.smithi088.stdout:(115/120): openblas-0.3.12-1.el8.x86_64.rpm 59 MB/s | 4.6 MB 00:00 2023-10-18T16:05:05.182 INFO:teuthology.orchestra.run.smithi088.stdout:(116/120): openblas-threads-0.3.12-1.el8.x86_64 51 MB/s | 4.7 MB 00:00 2023-10-18T16:05:05.189 INFO:teuthology.orchestra.run.smithi088.stdout:(117/120): automake-1.16.1-7.el8.noarch.rpm 34 MB/s | 713 kB 00:00 2023-10-18T16:05:05.837 INFO:teuthology.orchestra.run.smithi151.stdout:(10/120): ceph-osd-16.2.14-100.gb4ddd52c.el8.x8 5.8 MB/s | 18 MB 00:03 2023-10-18T16:05:05.952 INFO:teuthology.orchestra.run.smithi151.stdout:(11/120): cephfs-mirror-16.2.14-100.gb4ddd52c.e 2.2 MB/s | 257 kB 00:00 2023-10-18T16:05:06.011 INFO:teuthology.orchestra.run.smithi151.stdout:(12/120): libcephfs-devel-16.2.14-100.gb4ddd52c 432 kB/s | 25 kB 00:00 2023-10-18T16:05:06.351 INFO:teuthology.orchestra.run.smithi151.stdout:(13/120): libcephfs2-16.2.14-100.gb4ddd52c.el8. 2.4 MB/s | 825 kB 00:00 2023-10-18T16:05:06.417 INFO:teuthology.orchestra.run.smithi151.stdout:(14/120): libcephsqlite-16.2.14-100.gb4ddd52c.e 2.6 MB/s | 174 kB 00:00 2023-10-18T16:05:06.481 INFO:teuthology.orchestra.run.smithi151.stdout:(15/120): librados-devel-16.2.14-100.gb4ddd52c. 2.0 MB/s | 131 kB 00:00 2023-10-18T16:05:06.663 INFO:teuthology.orchestra.run.smithi151.stdout:(16/120): libradosstriper1-16.2.14-100.gb4ddd52 2.7 MB/s | 494 kB 00:00 2023-10-18T16:05:07.521 INFO:teuthology.orchestra.run.smithi151.stdout:(17/120): ceph-common-16.2.14-100.gb4ddd52c.el8 3.7 MB/s | 24 MB 00:06 2023-10-18T16:05:07.537 INFO:teuthology.orchestra.run.smithi088.stdout:(118/120): ceph-test-16.2.14-100.gb4ddd52c.el8. 6.8 MB/s | 52 MB 00:07 2023-10-18T16:05:07.593 INFO:teuthology.orchestra.run.smithi151.stdout:(18/120): python3-ceph-argparse-16.2.14-100.gb4 629 kB/s | 45 kB 00:00 2023-10-18T16:05:07.627 INFO:teuthology.orchestra.run.smithi151.stdout:(19/120): librgw2-16.2.14-100.gb4ddd52c.el8.x86 3.9 MB/s | 3.8 MB 00:00 2023-10-18T16:05:07.650 INFO:teuthology.orchestra.run.smithi151.stdout:(20/120): python3-ceph-common-16.2.14-100.gb4dd 1.7 MB/s | 99 kB 00:00 2023-10-18T16:05:07.651 INFO:teuthology.orchestra.run.smithi151.stdout:(21/120): python3-cephfs-16.2.14-100.gb4ddd52c. 8.9 MB/s | 215 kB 00:00 2023-10-18T16:05:07.700 INFO:teuthology.orchestra.run.smithi151.stdout:(22/120): python3-rbd-16.2.14-100.gb4ddd52c.el8 7.4 MB/s | 369 kB 00:00 2023-10-18T16:05:07.723 INFO:teuthology.orchestra.run.smithi151.stdout:(23/120): python3-rgw-16.2.14-100.gb4ddd52c.el8 5.1 MB/s | 114 kB 00:00 2023-10-18T16:05:07.745 INFO:teuthology.orchestra.run.smithi151.stdout:(24/120): rbd-fuse-16.2.14-100.gb4ddd52c.el8.x8 3.9 MB/s | 88 kB 00:00 2023-10-18T16:05:07.770 INFO:teuthology.orchestra.run.smithi151.stdout:(25/120): python3-rados-16.2.14-100.gb4ddd52c.e 3.2 MB/s | 390 kB 00:00 2023-10-18T16:05:07.824 INFO:teuthology.orchestra.run.smithi151.stdout:(26/120): rbd-nbd-16.2.14-100.gb4ddd52c.el8.x86 3.1 MB/s | 173 kB 00:00 2023-10-18T16:05:07.837 INFO:teuthology.orchestra.run.smithi088.stdout:(119/120): librados2-16.2.14-100.gb4ddd52c.el8. 1.4 MB/s | 3.7 MB 00:02 2023-10-18T16:05:07.863 INFO:teuthology.orchestra.run.smithi151.stdout:(27/120): ceph-grafana-dashboards-16.2.14-100.g 636 kB/s | 24 kB 00:00 2023-10-18T16:05:07.913 INFO:teuthology.orchestra.run.smithi151.stdout:(28/120): ceph-mgr-cephadm-16.2.14-100.gb4ddd52 2.3 MB/s | 116 kB 00:00 2023-10-18T16:05:08.245 INFO:teuthology.orchestra.run.smithi151.stdout:(29/120): rbd-mirror-16.2.14-100.gb4ddd52c.el8. 7.3 MB/s | 3.7 MB 00:00 2023-10-18T16:05:08.404 INFO:teuthology.orchestra.run.smithi151.stdout:(30/120): ceph-mgr-dashboard-16.2.14-100.gb4ddd 6.4 MB/s | 3.1 MB 00:00 2023-10-18T16:05:08.459 INFO:teuthology.orchestra.run.smithi151.stdout:(31/120): ceph-mgr-modules-core-16.2.14-100.gb4 4.4 MB/s | 244 kB 00:00 2023-10-18T16:05:08.503 INFO:teuthology.orchestra.run.smithi151.stdout:(32/120): ceph-mgr-rook-16.2.14-100.gb4ddd52c.e 628 kB/s | 27 kB 00:00 2023-10-18T16:05:08.545 INFO:teuthology.orchestra.run.smithi151.stdout:(33/120): ceph-prometheus-alerts-16.2.14-100.gb 354 kB/s | 15 kB 00:00 2023-10-18T16:05:08.591 INFO:teuthology.orchestra.run.smithi151.stdout:(34/120): cephadm-16.2.14-100.gb4ddd52c.el8.noa 1.8 MB/s | 84 kB 00:00 2023-10-18T16:05:08.635 INFO:teuthology.orchestra.run.smithi151.stdout:(35/120): cephfs-top-16.2.14-100.gb4ddd52c.el8. 431 kB/s | 19 kB 00:00 2023-10-18T16:05:09.090 INFO:teuthology.orchestra.run.smithi151.stdout:(36/120): ceph-test-16.2.14-100.gb4ddd52c.el8.x 11 MB/s | 52 MB 00:04 2023-10-18T16:05:09.478 INFO:teuthology.orchestra.run.smithi151.stdout:(37/120): ceph-mgr-diskprediction-local-16.2.14 7.0 MB/s | 8.6 MB 00:01 2023-10-18T16:05:09.615 INFO:teuthology.orchestra.run.smithi151.stdout:(38/120): leveldb-1.22-1.el8.x86_64.rpm 344 kB/s | 181 kB 00:00 2023-10-18T16:05:09.641 INFO:teuthology.orchestra.run.smithi088.stdout:(120/120): librbd1-16.2.14-100.gb4ddd52c.el8.x8 961 kB/s | 4.2 MB 00:04 2023-10-18T16:05:09.646 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2023-10-18T16:05:09.647 INFO:teuthology.orchestra.run.smithi088.stdout:Total 14 MB/s | 201 MB 00:13 2023-10-18T16:05:09.667 INFO:teuthology.orchestra.run.smithi151.stdout:(39/120): gperftools-libs-2.7-9.el8.x86_64.rpm 297 kB/s | 306 kB 00:01 2023-10-18T16:05:09.698 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2023-10-18T16:05:09.716 INFO:teuthology.orchestra.run.smithi151.stdout:(40/120): libunwind-1.3.1-3.el8.x86_64.rpm 744 kB/s | 75 kB 00:00 2023-10-18T16:05:09.760 INFO:teuthology.orchestra.run.smithi151.stdout:(41/120): python3-apipkg-1.5-6.el8.noarch.rpm 227 kB/s | 21 kB 00:00 2023-10-18T16:05:09.808 INFO:teuthology.orchestra.run.smithi151.stdout:(42/120): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 475 kB/s | 44 kB 00:00 2023-10-18T16:05:09.814 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2023-10-18T16:05:09.814 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2023-10-18T16:05:09.871 INFO:teuthology.orchestra.run.smithi151.stdout:(43/120): python3-beautifulsoup4-4.6.3-2.el8.1. 1.6 MB/s | 185 kB 00:00 2023-10-18T16:05:09.872 INFO:teuthology.orchestra.run.smithi151.stdout:(44/120): liboath-2.6.2-3.el8.x86_64.rpm 149 kB/s | 59 kB 00:00 2023-10-18T16:05:09.898 INFO:teuthology.orchestra.run.smithi151.stdout:(45/120): python3-cachetools-3.1.1-4.el8.noarch 367 kB/s | 33 kB 00:00 2023-10-18T16:05:09.957 INFO:teuthology.orchestra.run.smithi151.stdout:(46/120): python3-certifi-2018.10.15-7.el8.noar 190 kB/s | 16 kB 00:00 2023-10-18T16:05:10.115 INFO:teuthology.orchestra.run.smithi151.stdout:(47/120): python3-cheroot-8.5.2-1.el8.noarch.rp 711 kB/s | 173 kB 00:00 2023-10-18T16:05:10.132 INFO:teuthology.orchestra.run.smithi151.stdout:(48/120): python3-execnet-1.7.1-1.el8.noarch.rp 2.0 MB/s | 355 kB 00:00 2023-10-18T16:05:10.137 INFO:teuthology.orchestra.run.smithi151.stdout:(49/120): python3-cherrypy-18.4.0-1.el8.noarch. 1.6 MB/s | 384 kB 00:00 2023-10-18T16:05:10.218 INFO:teuthology.orchestra.run.smithi151.stdout:(50/120): python3-jaraco-6.2-6.el8.noarch.rpm 130 kB/s | 11 kB 00:00 2023-10-18T16:05:10.225 INFO:teuthology.orchestra.run.smithi151.stdout:(51/120): python3-jaraco-functools-2.0-4.el8.no 201 kB/s | 18 kB 00:00 2023-10-18T16:05:10.229 INFO:teuthology.orchestra.run.smithi151.stdout:(52/120): python3-google-auth-1.1.1-10.el8.noar 717 kB/s | 81 kB 00:00 2023-10-18T16:05:10.324 INFO:teuthology.orchestra.run.smithi151.stdout:(53/120): python3-logutils-0.3.5-11.el8.noarch. 494 kB/s | 49 kB 00:00 2023-10-18T16:05:10.340 INFO:teuthology.orchestra.run.smithi151.stdout:(54/120): python3-more-itertools-7.2.0-3.el8.no 532 kB/s | 59 kB 00:00 2023-10-18T16:05:10.422 INFO:teuthology.orchestra.run.smithi151.stdout:(55/120): python3-portend-2.6-1.el8.noarch.rpm 190 kB/s | 16 kB 00:00 2023-10-18T16:05:10.445 INFO:teuthology.orchestra.run.smithi151.stdout:(56/120): python3-pecan-1.3.2-9.el8.noarch.rpm 2.3 MB/s | 283 kB 00:00 2023-10-18T16:05:10.478 INFO:teuthology.orchestra.run.smithi151.stdout:(57/120): python3-kubernetes-11.0.0-6.el8.noarc 4.5 MB/s | 1.2 MB 00:00 2023-10-18T16:05:10.529 INFO:teuthology.orchestra.run.smithi151.stdout:(58/120): python3-repoze-lru-0.7-6.el8.noarch.r 403 kB/s | 34 kB 00:00 2023-10-18T16:05:10.532 INFO:teuthology.orchestra.run.smithi151.stdout:(59/120): python3-remoto-1.2.1-1.el8.noarch.rpm 442 kB/s | 49 kB 00:00 2023-10-18T16:05:10.564 INFO:teuthology.orchestra.run.smithi151.stdout:(60/120): python3-routes-2.4.1-12.el8.noarch.rp 2.2 MB/s | 196 kB 00:00 2023-10-18T16:05:10.617 INFO:teuthology.orchestra.run.smithi151.stdout:(61/120): python3-rsa-4.9-2.el8.noarch.rpm 732 kB/s | 64 kB 00:00 2023-10-18T16:05:10.617 INFO:teuthology.orchestra.run.smithi151.stdout:(62/120): python3-simplegeneric-0.8.1-17.el8.no 221 kB/s | 19 kB 00:00 2023-10-18T16:05:10.642 INFO:teuthology.orchestra.run.smithi151.stdout:(63/120): python3-singledispatch-3.4.0.3-18.el8 307 kB/s | 24 kB 00:00 2023-10-18T16:05:10.700 INFO:teuthology.orchestra.run.smithi151.stdout:(64/120): python3-tempora-1.14.1-5.el8.noarch.r 353 kB/s | 29 kB 00:00 2023-10-18T16:05:10.705 INFO:teuthology.orchestra.run.smithi151.stdout:(65/120): python3-trustme-0.6.0-4.el8.noarch.rp 315 kB/s | 27 kB 00:00 2023-10-18T16:05:10.731 INFO:teuthology.orchestra.run.smithi151.stdout:(66/120): python3-waitress-1.4.3-1.el8.noarch.r 2.8 MB/s | 255 kB 00:00 2023-10-18T16:05:10.807 INFO:teuthology.orchestra.run.smithi151.stdout:(67/120): python3-websocket-client-0.56.0-5.el8 601 kB/s | 61 kB 00:00 2023-10-18T16:05:10.817 INFO:teuthology.orchestra.run.smithi151.stdout:(68/120): python3-webtest-2.0.33-1.el8.noarch.r 1.0 MB/s | 85 kB 00:00 2023-10-18T16:05:10.824 INFO:teuthology.orchestra.run.smithi151.stdout:(69/120): python3-webob-1.8.5-1.el8.1.noarch.rp 2.0 MB/s | 251 kB 00:00 2023-10-18T16:05:10.825 INFO:teuthology.orchestra.run.smithi151.stdout:(70/120): xmlstarlet-1.6.1-20.el8.x86_64.rpm 8.4 MB/s | 70 kB 00:00 2023-10-18T16:05:10.854 INFO:teuthology.orchestra.run.smithi151.stdout:(71/120): python3-jwt-1.6.1-2.el8.noarch.rpm 1.5 MB/s | 43 kB 00:00 2023-10-18T16:05:10.863 INFO:teuthology.orchestra.run.smithi151.stdout:(72/120): libaio-devel-0.3.112-1.el8.x86_64.rpm 487 kB/s | 19 kB 00:00 2023-10-18T16:05:10.875 INFO:teuthology.orchestra.run.smithi151.stdout:(73/120): python3-oauthlib-2.1.0-1.el8.noarch.r 7.1 MB/s | 155 kB 00:00 2023-10-18T16:05:10.880 INFO:teuthology.orchestra.run.smithi151.stdout:(74/120): pcre2-utf32-10.32-2.el8.x86_64.rpm 13 MB/s | 220 kB 00:00 2023-10-18T16:05:10.892 INFO:teuthology.orchestra.run.smithi151.stdout:(75/120): python3-zc-lockfile-2.0-2.el8.noarch. 269 kB/s | 23 kB 00:00 2023-10-18T16:05:10.895 INFO:teuthology.orchestra.run.smithi151.stdout:(76/120): keyutils-libs-devel-1.5.10-6.el8.x86_ 2.4 MB/s | 48 kB 00:00 2023-10-18T16:05:10.906 INFO:teuthology.orchestra.run.smithi151.stdout:(77/120): pcre2-utf16-10.32-2.el8.x86_64.rpm 8.6 MB/s | 229 kB 00:00 2023-10-18T16:05:10.927 INFO:teuthology.orchestra.run.smithi151.stdout:(78/120): python3-requests-oauthlib-1.0.0-1.el8 1.2 MB/s | 43 kB 00:00 2023-10-18T16:05:10.928 INFO:teuthology.orchestra.run.smithi151.stdout:(79/120): libverto-devel-0.3.0-5.el8.x86_64.rpm 1.1 MB/s | 18 kB 00:00 2023-10-18T16:05:10.941 INFO:teuthology.orchestra.run.smithi151.stdout:(80/120): openssl-devel-1.1.1g-15.el8_3.x86_64. 50 MB/s | 2.3 MB 00:00 2023-10-18T16:05:10.946 INFO:teuthology.orchestra.run.smithi151.stdout:(81/120): fuse-2.9.7-12.el8.x86_64.rpm 4.5 MB/s | 83 kB 00:00 2023-10-18T16:05:10.949 INFO:teuthology.orchestra.run.smithi151.stdout:(82/120): pcre2-devel-10.32-2.el8.x86_64.rpm 27 MB/s | 605 kB 00:00 2023-10-18T16:05:10.958 INFO:teuthology.orchestra.run.smithi151.stdout:(83/120): elfutils-libelf-devel-0.182-3.el8.x86 3.5 MB/s | 59 kB 00:00 2023-10-18T16:05:10.965 INFO:teuthology.orchestra.run.smithi151.stdout:(84/120): libquadmath-8.4.1-1.el8.x86_64.rpm 9.0 MB/s | 169 kB 00:00 2023-10-18T16:05:10.969 INFO:teuthology.orchestra.run.smithi151.stdout:(85/120): libcom_err-devel-1.45.6-1.el8.x86_64. 1.9 MB/s | 38 kB 00:00 2023-10-18T16:05:10.981 INFO:teuthology.orchestra.run.smithi151.stdout:(86/120): libsepol-devel-2.9-2.el8.x86_64.rpm 3.7 MB/s | 86 kB 00:00 2023-10-18T16:05:10.983 INFO:teuthology.orchestra.run.smithi151.stdout:(87/120): zlib-devel-1.2.11-17.el8.x86_64.rpm 3.1 MB/s | 58 kB 00:00 2023-10-18T16:05:10.989 INFO:teuthology.orchestra.run.smithi151.stdout:(88/120): librabbitmq-0.9.0-3.el8.x86_64.rpm 2.4 MB/s | 47 kB 00:00 2023-10-18T16:05:11.008 INFO:teuthology.orchestra.run.smithi151.stdout:(89/120): krb5-devel-1.18.2-8.3.el8_4.x86_64.rp 20 MB/s | 559 kB 00:00 2023-10-18T16:05:11.009 INFO:teuthology.orchestra.run.smithi151.stdout:(90/120): xfsprogs-devel-5.0.0-8.el8.x86_64.rpm 3.4 MB/s | 70 kB 00:00 2023-10-18T16:05:11.012 INFO:teuthology.orchestra.run.smithi151.stdout:(91/120): libgfortran-8.4.1-1.el8.x86_64.rpm 21 MB/s | 639 kB 00:00 2023-10-18T16:05:11.028 INFO:teuthology.orchestra.run.smithi151.stdout:(92/120): libuuid-devel-2.32.1-27.el8.x86_64.rp 5.2 MB/s | 97 kB 00:00 2023-10-18T16:05:11.029 INFO:teuthology.orchestra.run.smithi151.stdout:(93/120): libselinux-devel-2.9-5.el8.x86_64.rpm 9.4 MB/s | 200 kB 00:00 2023-10-18T16:05:11.045 INFO:teuthology.orchestra.run.smithi151.stdout:(94/120): python3-prettytable-0.7.2-14.el8.noar 2.6 MB/s | 44 kB 00:00 2023-10-18T16:05:11.045 INFO:teuthology.orchestra.run.smithi151.stdout:(95/120): python3-jsonpatch-1.21-2.el8.noarch.r 1.6 MB/s | 27 kB 00:00 2023-10-18T16:05:11.061 INFO:teuthology.orchestra.run.smithi151.stdout:(96/120): jq-1.5-12.el8.x86_64.rpm 9.5 MB/s | 161 kB 00:00 2023-10-18T16:05:11.067 INFO:teuthology.orchestra.run.smithi151.stdout:(97/120): flex-2.6.1-9.el8.x86_64.rpm 15 MB/s | 320 kB 00:00 2023-10-18T16:05:11.083 INFO:teuthology.orchestra.run.smithi151.stdout:(98/120): python3-werkzeug-0.12.2-4.el8.noarch. 21 MB/s | 457 kB 00:00 2023-10-18T16:05:11.102 INFO:teuthology.orchestra.run.smithi151.stdout:(99/120): python3-markupsafe-0.23-19.el8.x86_64 2.1 MB/s | 39 kB 00:00 2023-10-18T16:05:11.128 INFO:teuthology.orchestra.run.smithi151.stdout:(100/120): bison-3.0.4-10.el8.x86_64.rpm 26 MB/s | 688 kB 00:00 2023-10-18T16:05:11.153 INFO:teuthology.orchestra.run.smithi151.stdout:(101/120): python3-babel-2.5.1-5.el8.noarch.rpm 56 MB/s | 4.8 MB 00:00 2023-10-18T16:05:11.157 INFO:teuthology.orchestra.run.smithi151.stdout:(102/120): lttng-ust-2.8.1-11.el8.x86_64.rpm 9.0 MB/s | 259 kB 00:00 2023-10-18T16:05:11.173 INFO:teuthology.orchestra.run.smithi151.stdout:(103/120): python3-numpy-f2py-1.14.3-9.el8.x86_ 11 MB/s | 225 kB 00:00 2023-10-18T16:05:11.175 INFO:teuthology.orchestra.run.smithi151.stdout:(104/120): python3-pyasn1-0.3.7-6.el8.noarch.rp 6.8 MB/s | 126 kB 00:00 2023-10-18T16:05:11.317 INFO:teuthology.orchestra.run.smithi151.stdout:(105/120): python3-scipy-1.0.0-20.module+el8.1. 48 MB/s | 15 MB 00:00 2023-10-18T16:05:11.318 INFO:teuthology.orchestra.run.smithi151.stdout:(106/120): oniguruma-6.8.2-2.el8.x86_64.rpm 1.3 MB/s | 187 kB 00:00 2023-10-18T16:05:11.340 INFO:teuthology.orchestra.run.smithi151.stdout:(107/120): python3-jinja2-2.10.1-2.el8_0.noarch 22 MB/s | 538 kB 00:00 2023-10-18T16:05:11.357 INFO:teuthology.orchestra.run.smithi151.stdout:(108/120): python3-numpy-1.14.3-9.el8.x86_64.rp 20 MB/s | 3.7 MB 00:00 2023-10-18T16:05:11.360 INFO:teuthology.orchestra.run.smithi151.stdout:(109/120): socat-1.7.3.3-2.el8.x86_64.rpm 7.1 MB/s | 302 kB 00:00 2023-10-18T16:05:11.362 INFO:teuthology.orchestra.run.smithi151.stdout:(110/120): python3-mako-1.0.6-13.el8.noarch.rpm 7.1 MB/s | 157 kB 00:00 2023-10-18T16:05:11.378 INFO:teuthology.orchestra.run.smithi151.stdout:(111/120): python3-jsonpointer-1.10-11.el8.noar 1.2 MB/s | 20 kB 00:00 2023-10-18T16:05:11.382 INFO:teuthology.orchestra.run.smithi151.stdout:(112/120): librdkafka-0.11.4-1.el8.x86_64.rpm 16 MB/s | 353 kB 00:00 2023-10-18T16:05:11.383 INFO:teuthology.orchestra.run.smithi151.stdout:(113/120): python3-pyasn1-modules-0.3.7-6.el8.n 4.2 MB/s | 110 kB 00:00 2023-10-18T16:05:11.399 INFO:teuthology.orchestra.run.smithi151.stdout:(114/120): python3-pytz-2017.2-9.el8.noarch.rpm 3.2 MB/s | 54 kB 00:00 2023-10-18T16:05:11.403 INFO:teuthology.orchestra.run.smithi151.stdout:(115/120): libtool-2.4.6-25.el8.x86_64.rpm 28 MB/s | 709 kB 00:00 2023-10-18T16:05:11.432 INFO:teuthology.orchestra.run.smithi151.stdout:(116/120): automake-1.16.1-7.el8.noarch.rpm 24 MB/s | 713 kB 00:00 2023-10-18T16:05:11.464 INFO:teuthology.orchestra.run.smithi151.stdout:(117/120): openblas-threads-0.3.12-1.el8.x86_64 59 MB/s | 4.7 MB 00:00 2023-10-18T16:05:11.471 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2023-10-18T16:05:11.475 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2023-10-18T16:05:11.486 INFO:teuthology.orchestra.run.smithi151.stdout:(118/120): openblas-0.3.12-1.el8.x86_64.rpm 52 MB/s | 4.6 MB 00:00 2023-10-18T16:05:11.859 INFO:teuthology.orchestra.run.smithi151.stdout:(119/120): librados2-16.2.14-100.gb4ddd52c.el8. 8.8 MB/s | 3.7 MB 00:00 2023-10-18T16:05:11.889 INFO:teuthology.orchestra.run.smithi151.stdout:(120/120): librbd1-16.2.14-100.gb4ddd52c.el8.x8 9.8 MB/s | 4.2 MB 00:00 2023-10-18T16:05:11.894 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2023-10-18T16:05:11.895 INFO:teuthology.orchestra.run.smithi151.stdout:Total 18 MB/s | 201 MB 00:11 2023-10-18T16:05:11.942 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2023-10-18T16:05:12.055 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2023-10-18T16:05:12.055 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2023-10-18T16:05:13.851 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2023-10-18T16:05:13.855 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2023-10-18T16:05:16.589 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2023-10-18T16:05:16.809 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/122 2023-10-18T16:05:17.000 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : leveldb-1.22-1.el8.x86_64 2/122 2023-10-18T16:05:17.169 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 3/122 2023-10-18T16:05:17.306 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 3/122 2023-10-18T16:05:17.493 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/122 2023-10-18T16:05:17.904 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/122 2023-10-18T16:05:18.641 INFO:teuthology.orchestra.run.smithi088.stdout: Upgrading : librados2-2:16.2.14-100.gb4ddd52c.el8.x86_64 5/122 2023-10-18T16:05:18.664 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2023-10-18T16:05:18.860 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/122 2023-10-18T16:05:19.051 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librados2-2:16.2.14-100.gb4ddd52c.el8.x86_64 5/122 2023-10-18T16:05:19.103 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : leveldb-1.22-1.el8.x86_64 2/122 2023-10-18T16:05:19.243 INFO:teuthology.orchestra.run.smithi088.stdout: Upgrading : librbd1-2:16.2.14-100.gb4ddd52c.el8.x86_64 6/122 2023-10-18T16:05:19.296 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 3/122 2023-10-18T16:05:19.356 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librbd1-2:16.2.14-100.gb4ddd52c.el8.x86_64 6/122 2023-10-18T16:05:19.459 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 3/122 2023-10-18T16:05:19.584 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libcephfs2-2:16.2.14-100.gb4ddd52c.el8.x86_64 7/122 2023-10-18T16:05:19.638 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/122 2023-10-18T16:05:19.686 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libcephfs2-2:16.2.14-100.gb4ddd52c.el8.x86_64 7/122 2023-10-18T16:05:19.881 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-rados-2:16.2.14-100.gb4ddd52c.el8.x86_64 8/122 2023-10-18T16:05:20.047 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/122 2023-10-18T16:05:20.271 INFO:teuthology.orchestra.run.smithi151.stdout: Upgrading : librados2-2:16.2.14-100.gb4ddd52c.el8.x86_64 5/122 2023-10-18T16:05:20.338 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 9/122 2023-10-18T16:05:20.526 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : librgw2-2:16.2.14-100.gb4ddd52c.el8.x86_64 10/122 2023-10-18T16:05:20.644 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librgw2-2:16.2.14-100.gb4ddd52c.el8.x86_64 10/122 2023-10-18T16:05:20.683 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librados2-2:16.2.14-100.gb4ddd52c.el8.x86_64 5/122 2023-10-18T16:05:20.880 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/122 2023-10-18T16:05:20.921 INFO:teuthology.orchestra.run.smithi151.stdout: Upgrading : librbd1-2:16.2.14-100.gb4ddd52c.el8.x86_64 6/122 2023-10-18T16:05:21.041 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/122 2023-10-18T16:05:21.050 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librbd1-2:16.2.14-100.gb4ddd52c.el8.x86_64 6/122 2023-10-18T16:05:21.218 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libcephsqlite-2:16.2.14-100.gb4ddd52c.el8.x86_64 13/122 2023-10-18T16:05:21.270 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libcephfs2-2:16.2.14-100.gb4ddd52c.el8.x86_64 7/122 2023-10-18T16:05:21.351 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libcephsqlite-2:16.2.14-100.gb4ddd52c.el8.x86_64 13/122 2023-10-18T16:05:21.371 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: libcephfs2-2:16.2.14-100.gb4ddd52c.el8.x86_64 7/122 2023-10-18T16:05:21.517 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libradosstriper1-2:16.2.14-100.gb4ddd52c.el8.x86 14/122 2023-10-18T16:05:21.580 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libradosstriper1-2:16.2.14-100.gb4ddd52c.el8.x86 14/122 2023-10-18T16:05:21.755 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 15/122 2023-10-18T16:05:22.015 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 16/122 2023-10-18T16:05:22.127 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-rados-2:16.2.14-100.gb4ddd52c.el8.x86_64 8/122 2023-10-18T16:05:22.242 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 17/122 2023-10-18T16:05:22.376 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : zlib-devel-1.2.11-17.el8.x86_64 18/122 2023-10-18T16:05:22.509 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 19/122 2023-10-18T16:05:22.619 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 9/122 2023-10-18T16:05:22.645 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 19/122 2023-10-18T16:05:22.771 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : librgw2-2:16.2.14-100.gb4ddd52c.el8.x86_64 10/122 2023-10-18T16:05:22.792 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 20/122 2023-10-18T16:05:22.878 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librgw2-2:16.2.14-100.gb4ddd52c.el8.x86_64 10/122 2023-10-18T16:05:23.125 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/122 2023-10-18T16:05:23.317 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/122 2023-10-18T16:05:23.339 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 20/122 2023-10-18T16:05:23.487 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libcephsqlite-2:16.2.14-100.gb4ddd52c.el8.x86_64 13/122 2023-10-18T16:05:23.500 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 21/122 2023-10-18T16:05:23.592 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: libcephsqlite-2:16.2.14-100.gb4ddd52c.el8.x86_64 13/122 2023-10-18T16:05:23.770 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libradosstriper1-2:16.2.14-100.gb4ddd52c.el8.x86 14/122 2023-10-18T16:05:23.851 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: libradosstriper1-2:16.2.14-100.gb4ddd52c.el8.x86 14/122 2023-10-18T16:05:24.035 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 21/122 2023-10-18T16:05:24.069 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 15/122 2023-10-18T16:05:24.217 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : openblas-0.3.12-1.el8.x86_64 22/122 2023-10-18T16:05:24.321 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 16/122 2023-10-18T16:05:24.578 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 17/122 2023-10-18T16:05:24.770 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : zlib-devel-1.2.11-17.el8.x86_64 18/122 2023-10-18T16:05:24.904 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 19/122 2023-10-18T16:05:24.950 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 22/122 2023-10-18T16:05:25.050 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 19/122 2023-10-18T16:05:25.204 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 23/122 2023-10-18T16:05:25.220 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 20/122 2023-10-18T16:05:25.435 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 24/122 2023-10-18T16:05:25.664 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/122 2023-10-18T16:05:25.759 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 20/122 2023-10-18T16:05:25.856 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-ceph-common-2:16.2.14-100.gb4ddd52c.el8. 26/122 2023-10-18T16:05:25.945 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 21/122 2023-10-18T16:05:26.006 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-ceph-argparse-2:16.2.14-100.gb4ddd52c.el 27/122 2023-10-18T16:05:26.259 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cephfs-2:16.2.14-100.gb4ddd52c.el8.x86_6 28/122 2023-10-18T16:05:26.474 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 21/122 2023-10-18T16:05:26.489 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 29/122 2023-10-18T16:05:26.653 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : openblas-0.3.12-1.el8.x86_64 22/122 2023-10-18T16:05:26.791 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 30/122 2023-10-18T16:05:26.991 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 31/122 2023-10-18T16:05:27.381 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 22/122 2023-10-18T16:05:28.122 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 23/122 2023-10-18T16:05:28.416 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 24/122 2023-10-18T16:05:28.665 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/122 2023-10-18T16:05:28.835 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-ceph-common-2:16.2.14-100.gb4ddd52c.el8. 26/122 2023-10-18T16:05:29.015 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-ceph-argparse-2:16.2.14-100.gb4ddd52c.el 27/122 2023-10-18T16:05:29.080 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 31/122 2023-10-18T16:05:29.275 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cephfs-2:16.2.14-100.gb4ddd52c.el8.x86_6 28/122 2023-10-18T16:05:29.458 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 32/122 2023-10-18T16:05:29.528 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 29/122 2023-10-18T16:05:29.836 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 30/122 2023-10-18T16:05:30.119 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 31/122 2023-10-18T16:05:30.467 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/122 2023-10-18T16:05:30.807 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 34/122 2023-10-18T16:05:31.052 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 35/122 2023-10-18T16:05:31.297 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-rsa-4.9-2.el8.noarch 36/122 2023-10-18T16:05:31.495 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 37/122 2023-10-18T16:05:31.699 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-rgw-2:16.2.14-100.gb4ddd52c.el8.x86_64 38/122 2023-10-18T16:05:31.888 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-rbd-2:16.2.14-100.gb4ddd52c.el8.x86_64 39/122 2023-10-18T16:05:32.175 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : librados-devel-2:16.2.14-100.gb4ddd52c.el8.x86_6 40/122 2023-10-18T16:05:32.204 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 31/122 2023-10-18T16:05:32.389 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : automake-1.16.1-7.el8.noarch 41/122 2023-10-18T16:05:32.560 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 32/122 2023-10-18T16:05:32.572 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 42/122 2023-10-18T16:05:32.770 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 43/122 2023-10-18T16:05:33.002 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 44/122 2023-10-18T16:05:33.173 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 45/122 2023-10-18T16:05:33.242 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 45/122 2023-10-18T16:05:33.415 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : jq-1.5-12.el8.x86_64 46/122 2023-10-18T16:05:33.556 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/122 2023-10-18T16:05:33.577 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 47/122 2023-10-18T16:05:33.755 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libuuid-devel-2.32.1-27.el8.x86_64 48/122 2023-10-18T16:05:33.899 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 34/122 2023-10-18T16:05:33.949 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libsepol-devel-2.9-2.el8.x86_64 49/122 2023-10-18T16:05:34.116 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libcom_err-devel-1.45.6-1.el8.x86_64 50/122 2023-10-18T16:05:34.172 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 35/122 2023-10-18T16:05:34.275 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : fuse-2.9.7-12.el8.x86_64 51/122 2023-10-18T16:05:34.410 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 52/122 2023-10-18T16:05:34.416 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-rsa-4.9-2.el8.noarch 36/122 2023-10-18T16:05:34.614 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 37/122 2023-10-18T16:05:34.837 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-rgw-2:16.2.14-100.gb4ddd52c.el8.x86_64 38/122 2023-10-18T16:05:34.938 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 53/122 2023-10-18T16:05:35.027 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-rbd-2:16.2.14-100.gb4ddd52c.el8.x86_64 39/122 2023-10-18T16:05:35.076 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : keyutils-libs-devel-1.5.10-6.el8.x86_64 54/122 2023-10-18T16:05:35.323 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : librados-devel-2:16.2.14-100.gb4ddd52c.el8.x86_6 40/122 2023-10-18T16:05:35.382 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/122 2023-10-18T16:05:35.568 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : automake-1.16.1-7.el8.noarch 41/122 2023-10-18T16:05:35.637 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/122 2023-10-18T16:05:35.778 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 42/122 2023-10-18T16:05:35.906 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 57/122 2023-10-18T16:05:35.965 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 43/122 2023-10-18T16:05:36.160 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : krb5-devel-1.18.2-8.3.el8_4.x86_64 58/122 2023-10-18T16:05:36.189 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 44/122 2023-10-18T16:05:36.298 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 59/122 2023-10-18T16:05:36.352 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 45/122 2023-10-18T16:05:36.431 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 45/122 2023-10-18T16:05:36.469 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 60/122 2023-10-18T16:05:36.635 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : jq-1.5-12.el8.x86_64 46/122 2023-10-18T16:05:36.769 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 61/122 2023-10-18T16:05:36.792 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 47/122 2023-10-18T16:05:36.981 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 62/122 2023-10-18T16:05:36.985 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libuuid-devel-2.32.1-27.el8.x86_64 48/122 2023-10-18T16:05:37.140 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libsepol-devel-2.9-2.el8.x86_64 49/122 2023-10-18T16:05:37.168 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 63/122 2023-10-18T16:05:37.343 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 64/122 2023-10-18T16:05:37.349 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libcom_err-devel-1.45.6-1.el8.x86_64 50/122 2023-10-18T16:05:37.508 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 65/122 2023-10-18T16:05:37.554 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : fuse-2.9.7-12.el8.x86_64 51/122 2023-10-18T16:05:37.673 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 52/122 2023-10-18T16:05:37.706 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 66/122 2023-10-18T16:05:37.866 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 53/122 2023-10-18T16:05:37.924 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 67/122 2023-10-18T16:05:38.047 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : keyutils-libs-devel-1.5.10-6.el8.x86_64 54/122 2023-10-18T16:05:38.092 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 68/122 2023-10-18T16:05:38.253 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 69/122 2023-10-18T16:05:38.400 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/122 2023-10-18T16:05:38.504 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 70/122 2023-10-18T16:05:38.642 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/122 2023-10-18T16:05:38.692 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 71/122 2023-10-18T16:05:38.894 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 72/122 2023-10-18T16:05:38.899 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 57/122 2023-10-18T16:05:39.196 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-portend-2.6-1.el8.noarch 73/122 2023-10-18T16:05:39.212 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : krb5-devel-1.18.2-8.3.el8_4.x86_64 58/122 2023-10-18T16:05:39.394 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 59/122 2023-10-18T16:05:39.463 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 74/122 2023-10-18T16:05:39.558 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 60/122 2023-10-18T16:05:39.654 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 75/122 2023-10-18T16:05:39.854 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 61/122 2023-10-18T16:05:39.879 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 76/122 2023-10-18T16:05:40.077 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 62/122 2023-10-18T16:05:40.254 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 63/122 2023-10-18T16:05:40.433 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 64/122 2023-10-18T16:05:40.590 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 65/122 2023-10-18T16:05:40.825 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 66/122 2023-10-18T16:05:41.025 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 67/122 2023-10-18T16:05:41.069 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 77/122 2023-10-18T16:05:41.239 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 68/122 2023-10-18T16:05:41.362 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 78/122 2023-10-18T16:05:41.409 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 69/122 2023-10-18T16:05:41.598 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 79/122 2023-10-18T16:05:41.696 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 70/122 2023-10-18T16:05:41.926 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 80/122 2023-10-18T16:05:41.950 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 71/122 2023-10-18T16:05:42.134 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 72/122 2023-10-18T16:05:42.222 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 81/122 2023-10-18T16:05:42.434 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-portend-2.6-1.el8.noarch 73/122 2023-10-18T16:05:42.436 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 82/122 2023-10-18T16:05:42.686 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 83/122 2023-10-18T16:05:42.718 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 74/122 2023-10-18T16:05:42.903 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 75/122 2023-10-18T16:05:42.986 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 84/122 2023-10-18T16:05:43.167 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 76/122 2023-10-18T16:05:43.177 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 85/122 2023-10-18T16:05:43.364 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 86/122 2023-10-18T16:05:43.580 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 87/122 2023-10-18T16:05:44.351 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 77/122 2023-10-18T16:05:44.641 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 78/122 2023-10-18T16:05:44.903 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 79/122 2023-10-18T16:05:45.234 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 80/122 2023-10-18T16:05:45.508 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 81/122 2023-10-18T16:05:45.733 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 82/122 2023-10-18T16:05:45.980 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 83/122 2023-10-18T16:05:46.274 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 84/122 2023-10-18T16:05:46.456 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 85/122 2023-10-18T16:05:46.644 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 86/122 2023-10-18T16:05:46.868 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 87/122 2023-10-18T16:05:48.452 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-common-2:16.2.14-100.gb4ddd52c.el8.x86_64 88/122 2023-10-18T16:05:48.679 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-common-2:16.2.14-100.gb4ddd52c.el8.x86_64 88/122 2023-10-18T16:05:49.394 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-common-2:16.2.14-100.gb4ddd52c.el8.x86_64 88/122 2023-10-18T16:05:49.687 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-base-2:16.2.14-100.gb4ddd52c.el8.x86_64 89/122 2023-10-18T16:05:49.783 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-base-2:16.2.14-100.gb4ddd52c.el8.x86_64 89/122 2023-10-18T16:05:49.930 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-selinux-2:16.2.14-100.gb4ddd52c.el8.x86_64 90/122 2023-10-18T16:05:51.795 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-common-2:16.2.14-100.gb4ddd52c.el8.x86_64 88/122 2023-10-18T16:05:52.041 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-common-2:16.2.14-100.gb4ddd52c.el8.x86_64 88/122 2023-10-18T16:05:52.761 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-common-2:16.2.14-100.gb4ddd52c.el8.x86_64 88/122 2023-10-18T16:05:53.086 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-base-2:16.2.14-100.gb4ddd52c.el8.x86_64 89/122 2023-10-18T16:05:53.179 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-base-2:16.2.14-100.gb4ddd52c.el8.x86_64 89/122 2023-10-18T16:05:53.346 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-selinux-2:16.2.14-100.gb4ddd52c.el8.x86_64 90/122 2023-10-18T16:06:07.636 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-selinux-2:16.2.14-100.gb4ddd52c.el8.x86_64 90/122 2023-10-18T16:06:07.636 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /sys 2023-10-18T16:06:07.636 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /proc 2023-10-18T16:06:07.637 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /dev 2023-10-18T16:06:07.637 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /run 2023-10-18T16:06:07.637 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /mnt 2023-10-18T16:06:07.637 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /var/tmp 2023-10-18T16:06:07.637 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /home 2023-10-18T16:06:07.638 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /tmp 2023-10-18T16:06:07.638 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /dev 2023-10-18T16:06:07.638 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:06:07.999 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mds-2:16.2.14-100.gb4ddd52c.el8.x86_64 91/122 2023-10-18T16:06:08.433 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mds-2:16.2.14-100.gb4ddd52c.el8.x86_64 91/122 2023-10-18T16:06:08.582 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mon-2:16.2.14-100.gb4ddd52c.el8.x86_64 92/122 2023-10-18T16:06:10.068 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mon-2:16.2.14-100.gb4ddd52c.el8.x86_64 92/122 2023-10-18T16:06:10.257 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-osd-2:16.2.14-100.gb4ddd52c.el8.x86_64 93/122 2023-10-18T16:06:10.278 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-osd-2:16.2.14-100.gb4ddd52c.el8.x86_64 93/122 2023-10-18T16:06:10.824 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-selinux-2:16.2.14-100.gb4ddd52c.el8.x86_64 90/122 2023-10-18T16:06:10.824 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /sys 2023-10-18T16:06:10.824 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /proc 2023-10-18T16:06:10.824 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /dev 2023-10-18T16:06:10.825 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /run 2023-10-18T16:06:10.825 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /mnt 2023-10-18T16:06:10.825 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /var/tmp 2023-10-18T16:06:10.825 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /home 2023-10-18T16:06:10.825 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /tmp 2023-10-18T16:06:10.826 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /dev 2023-10-18T16:06:10.826 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-10-18T16:06:11.188 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mds-2:16.2.14-100.gb4ddd52c.el8.x86_64 91/122 2023-10-18T16:06:11.621 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mds-2:16.2.14-100.gb4ddd52c.el8.x86_64 91/122 2023-10-18T16:06:11.772 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mon-2:16.2.14-100.gb4ddd52c.el8.x86_64 92/122 2023-10-18T16:06:13.119 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: cephadm-2:16.2.14-100.gb4ddd52c.el8.noarch 94/122 2023-10-18T16:06:13.253 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mon-2:16.2.14-100.gb4ddd52c.el8.x86_64 92/122 2023-10-18T16:06:13.268 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : cephadm-2:16.2.14-100.gb4ddd52c.el8.noarch 94/122 2023-10-18T16:06:13.454 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-osd-2:16.2.14-100.gb4ddd52c.el8.x86_64 93/122 2023-10-18T16:06:13.475 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-osd-2:16.2.14-100.gb4ddd52c.el8.x86_64 93/122 2023-10-18T16:06:13.489 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-prometheus-alerts-2:16.2.14-100.gb4ddd52c.e 95/122 2023-10-18T16:06:13.709 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-grafana-dashboards-2:16.2.14-100.gb4ddd52c. 96/122 2023-10-18T16:06:13.897 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-cephadm-2:16.2.14-100.gb4ddd52c.el8.noa 97/122 2023-10-18T16:06:14.863 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.14-100.gb4ddd52c.el8.noa 97/122 2023-10-18T16:06:15.148 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-dashboard-2:16.2.14-100.gb4ddd52c.el8.n 98/122 2023-10-18T16:06:16.194 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.14-100.gb4ddd52c.el8.n 98/122 2023-10-18T16:06:16.350 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: cephadm-2:16.2.14-100.gb4ddd52c.el8.noarch 94/122 2023-10-18T16:06:16.397 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.14-100.gb4d 99/122 2023-10-18T16:06:16.485 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : cephadm-2:16.2.14-100.gb4ddd52c.el8.noarch 94/122 2023-10-18T16:06:16.494 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.14-100.gb4d 99/122 2023-10-18T16:06:16.694 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-prometheus-alerts-2:16.2.14-100.gb4ddd52c.e 95/122 2023-10-18T16:06:16.833 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-modules-core-2:16.2.14-100.gb4ddd52c.el 100/122 2023-10-18T16:06:16.873 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-grafana-dashboards-2:16.2.14-100.gb4ddd52c. 96/122 2023-10-18T16:06:16.998 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-2:16.2.14-100.gb4ddd52c.el8.x86_64 101/122 2023-10-18T16:06:17.025 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-2:16.2.14-100.gb4ddd52c.el8.x86_64 101/122 2023-10-18T16:06:17.062 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mgr-cephadm-2:16.2.14-100.gb4ddd52c.el8.noa 97/122 2023-10-18T16:06:17.155 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-rook-2:16.2.14-100.gb4ddd52c.el8.noarch 102/122 2023-10-18T16:06:17.166 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.14-100.gb4ddd52c.el8.noarch 102/122 2023-10-18T16:06:18.030 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.14-100.gb4ddd52c.el8.noa 97/122 2023-10-18T16:06:18.259 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-2:16.2.14-100.gb4ddd52c.el8.x86_64 103/122 2023-10-18T16:06:18.265 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mgr-dashboard-2:16.2.14-100.gb4ddd52c.el8.n 98/122 2023-10-18T16:06:18.456 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-radosgw-2:16.2.14-100.gb4ddd52c.el8.x86_64 104/122 2023-10-18T16:06:18.550 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-radosgw-2:16.2.14-100.gb4ddd52c.el8.x86_64 104/122 2023-10-18T16:06:18.680 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-immutable-object-cache-2:16.2.14-100.gb4ddd 105/122 2023-10-18T16:06:18.722 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.14-100.gb4ddd 105/122 2023-10-18T16:06:18.872 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : cephfs-mirror-2:16.2.14-100.gb4ddd52c.el8.x86_64 106/122 2023-10-18T16:06:19.219 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: cephfs-mirror-2:16.2.14-100.gb4ddd52c.el8.x86_64 106/122 2023-10-18T16:06:19.305 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.14-100.gb4ddd52c.el8.n 98/122 2023-10-18T16:06:19.380 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : rbd-mirror-2:16.2.14-100.gb4ddd52c.el8.x86_64 107/122 2023-10-18T16:06:19.504 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.14-100.gb4d 99/122 2023-10-18T16:06:19.600 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.14-100.gb4d 99/122 2023-10-18T16:06:19.987 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mgr-modules-core-2:16.2.14-100.gb4ddd52c.el 100/122 2023-10-18T16:06:20.215 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mgr-2:16.2.14-100.gb4ddd52c.el8.x86_64 101/122 2023-10-18T16:06:20.242 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mgr-2:16.2.14-100.gb4ddd52c.el8.x86_64 101/122 2023-10-18T16:06:20.379 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mgr-rook-2:16.2.14-100.gb4ddd52c.el8.noarch 102/122 2023-10-18T16:06:20.390 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.14-100.gb4ddd52c.el8.noarch 102/122 2023-10-18T16:06:21.502 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-2:16.2.14-100.gb4ddd52c.el8.x86_64 103/122 2023-10-18T16:06:21.715 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-radosgw-2:16.2.14-100.gb4ddd52c.el8.x86_64 104/122 2023-10-18T16:06:21.810 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-radosgw-2:16.2.14-100.gb4ddd52c.el8.x86_64 104/122 2023-10-18T16:06:21.987 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-immutable-object-cache-2:16.2.14-100.gb4ddd 105/122 2023-10-18T16:06:22.028 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.14-100.gb4ddd 105/122 2023-10-18T16:06:22.183 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : cephfs-mirror-2:16.2.14-100.gb4ddd52c.el8.x86_64 106/122 2023-10-18T16:06:22.527 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: cephfs-mirror-2:16.2.14-100.gb4ddd52c.el8.x86_64 106/122 2023-10-18T16:06:22.729 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : rbd-mirror-2:16.2.14-100.gb4ddd52c.el8.x86_64 107/122 2023-10-18T16:06:23.940 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: rbd-mirror-2:16.2.14-100.gb4ddd52c.el8.x86_64 107/122 2023-10-18T16:06:24.863 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-test-2:16.2.14-100.gb4ddd52c.el8.x86_64 108/122 2023-10-18T16:06:25.195 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : openssl-devel-1:1.1.1g-15.el8_3.x86_64 109/122 2023-10-18T16:06:25.413 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-fuse-2:16.2.14-100.gb4ddd52c.el8.x86_64 110/122 2023-10-18T16:06:25.652 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : xfsprogs-devel-5.0.0-8.el8.x86_64 111/122 2023-10-18T16:06:25.845 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libtool-2.4.6-25.el8.x86_64 112/122 2023-10-18T16:06:25.856 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 112/122 2023-10-18T16:06:26.008 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libcephfs-devel-2:16.2.14-100.gb4ddd52c.el8.x86_ 113/122 2023-10-18T16:06:26.141 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : elfutils-libelf-devel-0.182-3.el8.x86_64 114/122 2023-10-18T16:06:26.320 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : cephfs-top-2:16.2.14-100.gb4ddd52c.el8.noarch 115/122 2023-10-18T16:06:26.468 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : rbd-fuse-2:16.2.14-100.gb4ddd52c.el8.x86_64 116/122 2023-10-18T16:06:26.713 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : rbd-nbd-2:16.2.14-100.gb4ddd52c.el8.x86_64 117/122 2023-10-18T16:06:26.911 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : bison-3.0.4-10.el8.x86_64 118/122 2023-10-18T16:06:26.969 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 118/122 2023-10-18T16:06:27.170 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : flex-2.6.1-9.el8.x86_64 119/122 2023-10-18T16:06:27.178 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 119/122 2023-10-18T16:06:27.276 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: rbd-mirror-2:16.2.14-100.gb4ddd52c.el8.x86_64 107/122 2023-10-18T16:06:27.305 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 120/122 2023-10-18T16:06:27.305 INFO:teuthology.orchestra.run.smithi088.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 121/122 2023-10-18T16:06:27.514 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 121/122 2023-10-18T16:06:27.514 INFO:teuthology.orchestra.run.smithi088.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 122/122 2023-10-18T16:06:28.195 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-test-2:16.2.14-100.gb4ddd52c.el8.x86_64 108/122 2023-10-18T16:06:28.370 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 122/122 2023-10-18T16:06:28.371 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-2:16.2.14-100.gb4ddd52c.el8.x86_64 1/122 2023-10-18T16:06:28.371 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-base-2:16.2.14-100.gb4ddd52c.el8.x86_64 2/122 2023-10-18T16:06:28.371 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-common-2:16.2.14-100.gb4ddd52c.el8.x86_64 3/122 2023-10-18T16:06:28.371 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-fuse-2:16.2.14-100.gb4ddd52c.el8.x86_64 4/122 2023-10-18T16:06:28.372 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-immutable-object-cache-2:16.2.14-100.gb4ddd 5/122 2023-10-18T16:06:28.372 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mds-2:16.2.14-100.gb4ddd52c.el8.x86_64 6/122 2023-10-18T16:06:28.372 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-2:16.2.14-100.gb4ddd52c.el8.x86_64 7/122 2023-10-18T16:06:28.372 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mon-2:16.2.14-100.gb4ddd52c.el8.x86_64 8/122 2023-10-18T16:06:28.372 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-osd-2:16.2.14-100.gb4ddd52c.el8.x86_64 9/122 2023-10-18T16:06:28.372 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-radosgw-2:16.2.14-100.gb4ddd52c.el8.x86_64 10/122 2023-10-18T16:06:28.373 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-selinux-2:16.2.14-100.gb4ddd52c.el8.x86_64 11/122 2023-10-18T16:06:28.373 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-test-2:16.2.14-100.gb4ddd52c.el8.x86_64 12/122 2023-10-18T16:06:28.373 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : cephfs-mirror-2:16.2.14-100.gb4ddd52c.el8.x86_64 13/122 2023-10-18T16:06:28.373 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libcephfs-devel-2:16.2.14-100.gb4ddd52c.el8.x86_ 14/122 2023-10-18T16:06:28.373 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libcephfs2-2:16.2.14-100.gb4ddd52c.el8.x86_64 15/122 2023-10-18T16:06:28.374 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libcephsqlite-2:16.2.14-100.gb4ddd52c.el8.x86_64 16/122 2023-10-18T16:06:28.374 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librados-devel-2:16.2.14-100.gb4ddd52c.el8.x86_6 17/122 2023-10-18T16:06:28.374 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libradosstriper1-2:16.2.14-100.gb4ddd52c.el8.x86 18/122 2023-10-18T16:06:28.374 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librgw2-2:16.2.14-100.gb4ddd52c.el8.x86_64 19/122 2023-10-18T16:06:28.374 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-ceph-argparse-2:16.2.14-100.gb4ddd52c.el 20/122 2023-10-18T16:06:28.375 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-ceph-common-2:16.2.14-100.gb4ddd52c.el8. 21/122 2023-10-18T16:06:28.375 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cephfs-2:16.2.14-100.gb4ddd52c.el8.x86_6 22/122 2023-10-18T16:06:28.375 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-rados-2:16.2.14-100.gb4ddd52c.el8.x86_64 23/122 2023-10-18T16:06:28.375 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-rbd-2:16.2.14-100.gb4ddd52c.el8.x86_64 24/122 2023-10-18T16:06:28.375 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-rgw-2:16.2.14-100.gb4ddd52c.el8.x86_64 25/122 2023-10-18T16:06:28.376 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : rbd-fuse-2:16.2.14-100.gb4ddd52c.el8.x86_64 26/122 2023-10-18T16:06:28.376 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : rbd-mirror-2:16.2.14-100.gb4ddd52c.el8.x86_64 27/122 2023-10-18T16:06:28.376 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : rbd-nbd-2:16.2.14-100.gb4ddd52c.el8.x86_64 28/122 2023-10-18T16:06:28.376 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-grafana-dashboards-2:16.2.14-100.gb4ddd52c. 29/122 2023-10-18T16:06:28.376 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-cephadm-2:16.2.14-100.gb4ddd52c.el8.noa 30/122 2023-10-18T16:06:28.377 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-dashboard-2:16.2.14-100.gb4ddd52c.el8.n 31/122 2023-10-18T16:06:28.377 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.14-100.gb4d 32/122 2023-10-18T16:06:28.377 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-modules-core-2:16.2.14-100.gb4ddd52c.el 33/122 2023-10-18T16:06:28.377 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-rook-2:16.2.14-100.gb4ddd52c.el8.noarch 34/122 2023-10-18T16:06:28.377 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-prometheus-alerts-2:16.2.14-100.gb4ddd52c.e 35/122 2023-10-18T16:06:28.378 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : cephadm-2:16.2.14-100.gb4ddd52c.el8.noarch 36/122 2023-10-18T16:06:28.378 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : cephfs-top-2:16.2.14-100.gb4ddd52c.el8.noarch 37/122 2023-10-18T16:06:28.378 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 38/122 2023-10-18T16:06:28.378 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : leveldb-1.22-1.el8.x86_64 39/122 2023-10-18T16:06:28.378 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/122 2023-10-18T16:06:28.379 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/122 2023-10-18T16:06:28.379 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 42/122 2023-10-18T16:06:28.379 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/122 2023-10-18T16:06:28.379 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/122 2023-10-18T16:06:28.379 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/122 2023-10-18T16:06:28.379 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/122 2023-10-18T16:06:28.380 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/122 2023-10-18T16:06:28.380 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/122 2023-10-18T16:06:28.380 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 49/122 2023-10-18T16:06:28.380 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 50/122 2023-10-18T16:06:28.380 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 51/122 2023-10-18T16:06:28.381 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 52/122 2023-10-18T16:06:28.381 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 53/122 2023-10-18T16:06:28.381 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 54/122 2023-10-18T16:06:28.381 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 55/122 2023-10-18T16:06:28.381 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/122 2023-10-18T16:06:28.382 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/122 2023-10-18T16:06:28.382 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 58/122 2023-10-18T16:06:28.382 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 59/122 2023-10-18T16:06:28.382 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 60/122 2023-10-18T16:06:28.382 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 61/122 2023-10-18T16:06:28.383 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 62/122 2023-10-18T16:06:28.383 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 63/122 2023-10-18T16:06:28.383 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 64/122 2023-10-18T16:06:28.383 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 65/122 2023-10-18T16:06:28.383 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 66/122 2023-10-18T16:06:28.384 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 67/122 2023-10-18T16:06:28.384 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 68/122 2023-10-18T16:06:28.384 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 69/122 2023-10-18T16:06:28.384 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 70/122 2023-10-18T16:06:28.384 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/122 2023-10-18T16:06:28.385 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 72/122 2023-10-18T16:06:28.385 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 73/122 2023-10-18T16:06:28.385 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 74/122 2023-10-18T16:06:28.385 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/122 2023-10-18T16:06:28.385 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : keyutils-libs-devel-1.5.10-6.el8.x86_64 76/122 2023-10-18T16:06:28.385 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 77/122 2023-10-18T16:06:28.386 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 78/122 2023-10-18T16:06:28.386 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : openssl-devel-1:1.1.1g-15.el8_3.x86_64 79/122 2023-10-18T16:06:28.386 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 80/122 2023-10-18T16:06:28.386 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 81/122 2023-10-18T16:06:28.386 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 82/122 2023-10-18T16:06:28.387 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : elfutils-libelf-devel-0.182-3.el8.x86_64 83/122 2023-10-18T16:06:28.387 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 84/122 2023-10-18T16:06:28.387 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libcom_err-devel-1.45.6-1.el8.x86_64 85/122 2023-10-18T16:06:28.387 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libsepol-devel-2.9-2.el8.x86_64 86/122 2023-10-18T16:06:28.387 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : zlib-devel-1.2.11-17.el8.x86_64 87/122 2023-10-18T16:06:28.388 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 88/122 2023-10-18T16:06:28.388 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : krb5-devel-1.18.2-8.3.el8_4.x86_64 89/122 2023-10-18T16:06:28.388 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 90/122 2023-10-18T16:06:28.388 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : xfsprogs-devel-5.0.0-8.el8.x86_64 91/122 2023-10-18T16:06:28.388 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 92/122 2023-10-18T16:06:28.389 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libuuid-devel-2.32.1-27.el8.x86_64 93/122 2023-10-18T16:06:28.389 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 94/122 2023-10-18T16:06:28.389 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 95/122 2023-10-18T16:06:28.389 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 96/122 2023-10-18T16:06:28.389 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : jq-1.5-12.el8.x86_64 97/122 2023-10-18T16:06:28.390 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : flex-2.6.1-9.el8.x86_64 98/122 2023-10-18T16:06:28.390 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 99/122 2023-10-18T16:06:28.390 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 100/122 2023-10-18T16:06:28.390 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 101/122 2023-10-18T16:06:28.390 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : bison-3.0.4-10.el8.x86_64 102/122 2023-10-18T16:06:28.391 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 103/122 2023-10-18T16:06:28.391 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 104/122 2023-10-18T16:06:28.391 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 105/122 2023-10-18T16:06:28.391 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 106/122 2023-10-18T16:06:28.391 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 107/122 2023-10-18T16:06:28.392 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 108/122 2023-10-18T16:06:28.392 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 109/122 2023-10-18T16:06:28.392 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 110/122 2023-10-18T16:06:28.392 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 111/122 2023-10-18T16:06:28.392 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 112/122 2023-10-18T16:06:28.393 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 113/122 2023-10-18T16:06:28.393 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 114/122 2023-10-18T16:06:28.393 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 115/122 2023-10-18T16:06:28.393 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 116/122 2023-10-18T16:06:28.393 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 117/122 2023-10-18T16:06:28.394 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : automake-1.16.1-7.el8.noarch 118/122 2023-10-18T16:06:28.394 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librados2-2:16.2.14-100.gb4ddd52c.el8.x86_64 119/122 2023-10-18T16:06:28.394 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 120/122 2023-10-18T16:06:28.394 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librbd1-2:16.2.14-100.gb4ddd52c.el8.x86_64 121/122 2023-10-18T16:06:28.561 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : openssl-devel-1:1.1.1g-15.el8_3.x86_64 109/122 2023-10-18T16:06:28.779 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-fuse-2:16.2.14-100.gb4ddd52c.el8.x86_64 110/122 2023-10-18T16:06:29.060 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : xfsprogs-devel-5.0.0-8.el8.x86_64 111/122 2023-10-18T16:06:29.227 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libtool-2.4.6-25.el8.x86_64 112/122 2023-10-18T16:06:29.238 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 112/122 2023-10-18T16:06:29.432 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libcephfs-devel-2:16.2.14-100.gb4ddd52c.el8.x86_ 113/122 2023-10-18T16:06:29.565 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : elfutils-libelf-devel-0.182-3.el8.x86_64 114/122 2023-10-18T16:06:29.743 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : cephfs-top-2:16.2.14-100.gb4ddd52c.el8.noarch 115/122 2023-10-18T16:06:29.901 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : rbd-fuse-2:16.2.14-100.gb4ddd52c.el8.x86_64 116/122 2023-10-18T16:06:30.187 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : rbd-nbd-2:16.2.14-100.gb4ddd52c.el8.x86_64 117/122 2023-10-18T16:06:30.352 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : bison-3.0.4-10.el8.x86_64 118/122 2023-10-18T16:06:30.418 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 118/122 2023-10-18T16:06:30.485 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 122/122 2023-10-18T16:06:30.486 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:43 ago on Wed 18 Oct 2023 04:04:47 PM UTC. 2023-10-18T16:06:30.577 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : flex-2.6.1-9.el8.x86_64 119/122 2023-10-18T16:06:30.586 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 119/122 2023-10-18T16:06:30.704 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 120/122 2023-10-18T16:06:30.705 INFO:teuthology.orchestra.run.smithi151.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 121/122 2023-10-18T16:06:30.929 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 121/122 2023-10-18T16:06:30.930 INFO:teuthology.orchestra.run.smithi151.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 122/122 2023-10-18T16:06:31.836 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 122/122 2023-10-18T16:06:31.837 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-2:16.2.14-100.gb4ddd52c.el8.x86_64 1/122 2023-10-18T16:06:31.837 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-base-2:16.2.14-100.gb4ddd52c.el8.x86_64 2/122 2023-10-18T16:06:31.837 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-common-2:16.2.14-100.gb4ddd52c.el8.x86_64 3/122 2023-10-18T16:06:31.837 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-fuse-2:16.2.14-100.gb4ddd52c.el8.x86_64 4/122 2023-10-18T16:06:31.838 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-immutable-object-cache-2:16.2.14-100.gb4ddd 5/122 2023-10-18T16:06:31.838 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mds-2:16.2.14-100.gb4ddd52c.el8.x86_64 6/122 2023-10-18T16:06:31.838 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mgr-2:16.2.14-100.gb4ddd52c.el8.x86_64 7/122 2023-10-18T16:06:31.838 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mon-2:16.2.14-100.gb4ddd52c.el8.x86_64 8/122 2023-10-18T16:06:31.838 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-osd-2:16.2.14-100.gb4ddd52c.el8.x86_64 9/122 2023-10-18T16:06:31.839 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-radosgw-2:16.2.14-100.gb4ddd52c.el8.x86_64 10/122 2023-10-18T16:06:31.839 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-selinux-2:16.2.14-100.gb4ddd52c.el8.x86_64 11/122 2023-10-18T16:06:31.839 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-test-2:16.2.14-100.gb4ddd52c.el8.x86_64 12/122 2023-10-18T16:06:31.839 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : cephfs-mirror-2:16.2.14-100.gb4ddd52c.el8.x86_64 13/122 2023-10-18T16:06:31.839 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libcephfs-devel-2:16.2.14-100.gb4ddd52c.el8.x86_ 14/122 2023-10-18T16:06:31.840 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libcephfs2-2:16.2.14-100.gb4ddd52c.el8.x86_64 15/122 2023-10-18T16:06:31.840 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libcephsqlite-2:16.2.14-100.gb4ddd52c.el8.x86_64 16/122 2023-10-18T16:06:31.840 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librados-devel-2:16.2.14-100.gb4ddd52c.el8.x86_6 17/122 2023-10-18T16:06:31.840 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libradosstriper1-2:16.2.14-100.gb4ddd52c.el8.x86 18/122 2023-10-18T16:06:31.840 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librgw2-2:16.2.14-100.gb4ddd52c.el8.x86_64 19/122 2023-10-18T16:06:31.841 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-ceph-argparse-2:16.2.14-100.gb4ddd52c.el 20/122 2023-10-18T16:06:31.841 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-ceph-common-2:16.2.14-100.gb4ddd52c.el8. 21/122 2023-10-18T16:06:31.841 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cephfs-2:16.2.14-100.gb4ddd52c.el8.x86_6 22/122 2023-10-18T16:06:31.841 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-rados-2:16.2.14-100.gb4ddd52c.el8.x86_64 23/122 2023-10-18T16:06:31.841 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-rbd-2:16.2.14-100.gb4ddd52c.el8.x86_64 24/122 2023-10-18T16:06:31.841 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-rgw-2:16.2.14-100.gb4ddd52c.el8.x86_64 25/122 2023-10-18T16:06:31.842 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : rbd-fuse-2:16.2.14-100.gb4ddd52c.el8.x86_64 26/122 2023-10-18T16:06:31.842 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : rbd-mirror-2:16.2.14-100.gb4ddd52c.el8.x86_64 27/122 2023-10-18T16:06:31.842 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : rbd-nbd-2:16.2.14-100.gb4ddd52c.el8.x86_64 28/122 2023-10-18T16:06:31.842 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-grafana-dashboards-2:16.2.14-100.gb4ddd52c. 29/122 2023-10-18T16:06:31.842 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mgr-cephadm-2:16.2.14-100.gb4ddd52c.el8.noa 30/122 2023-10-18T16:06:31.851 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mgr-dashboard-2:16.2.14-100.gb4ddd52c.el8.n 31/122 2023-10-18T16:06:31.851 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.14-100.gb4d 32/122 2023-10-18T16:06:31.851 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mgr-modules-core-2:16.2.14-100.gb4ddd52c.el 33/122 2023-10-18T16:06:31.851 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mgr-rook-2:16.2.14-100.gb4ddd52c.el8.noarch 34/122 2023-10-18T16:06:31.851 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-prometheus-alerts-2:16.2.14-100.gb4ddd52c.e 35/122 2023-10-18T16:06:31.852 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : cephadm-2:16.2.14-100.gb4ddd52c.el8.noarch 36/122 2023-10-18T16:06:31.852 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : cephfs-top-2:16.2.14-100.gb4ddd52c.el8.noarch 37/122 2023-10-18T16:06:31.852 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 38/122 2023-10-18T16:06:31.852 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : leveldb-1.22-1.el8.x86_64 39/122 2023-10-18T16:06:31.852 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/122 2023-10-18T16:06:31.853 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/122 2023-10-18T16:06:31.853 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 42/122 2023-10-18T16:06:31.853 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/122 2023-10-18T16:06:31.853 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/122 2023-10-18T16:06:31.853 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/122 2023-10-18T16:06:31.854 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/122 2023-10-18T16:06:31.854 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/122 2023-10-18T16:06:31.854 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/122 2023-10-18T16:06:31.854 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 49/122 2023-10-18T16:06:31.854 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 50/122 2023-10-18T16:06:31.857 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 51/122 2023-10-18T16:06:31.857 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 52/122 2023-10-18T16:06:31.857 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 53/122 2023-10-18T16:06:31.858 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 54/122 2023-10-18T16:06:31.858 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 55/122 2023-10-18T16:06:31.858 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/122 2023-10-18T16:06:31.858 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/122 2023-10-18T16:06:31.859 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 58/122 2023-10-18T16:06:31.859 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 59/122 2023-10-18T16:06:31.859 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 60/122 2023-10-18T16:06:31.859 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 61/122 2023-10-18T16:06:31.860 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 62/122 2023-10-18T16:06:31.860 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 63/122 2023-10-18T16:06:31.860 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 64/122 2023-10-18T16:06:31.860 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 65/122 2023-10-18T16:06:31.861 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 66/122 2023-10-18T16:06:31.861 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 67/122 2023-10-18T16:06:31.861 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 68/122 2023-10-18T16:06:31.861 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 69/122 2023-10-18T16:06:31.862 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 70/122 2023-10-18T16:06:31.862 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/122 2023-10-18T16:06:31.862 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 72/122 2023-10-18T16:06:31.862 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 73/122 2023-10-18T16:06:31.862 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 74/122 2023-10-18T16:06:31.863 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/122 2023-10-18T16:06:31.863 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : keyutils-libs-devel-1.5.10-6.el8.x86_64 76/122 2023-10-18T16:06:31.863 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 77/122 2023-10-18T16:06:31.863 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 78/122 2023-10-18T16:06:31.864 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : openssl-devel-1:1.1.1g-15.el8_3.x86_64 79/122 2023-10-18T16:06:31.864 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 80/122 2023-10-18T16:06:31.864 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 81/122 2023-10-18T16:06:31.864 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 82/122 2023-10-18T16:06:31.865 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : elfutils-libelf-devel-0.182-3.el8.x86_64 83/122 2023-10-18T16:06:31.865 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 84/122 2023-10-18T16:06:31.865 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libcom_err-devel-1.45.6-1.el8.x86_64 85/122 2023-10-18T16:06:31.865 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libsepol-devel-2.9-2.el8.x86_64 86/122 2023-10-18T16:06:31.865 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : zlib-devel-1.2.11-17.el8.x86_64 87/122 2023-10-18T16:06:31.866 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 88/122 2023-10-18T16:06:31.866 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : krb5-devel-1.18.2-8.3.el8_4.x86_64 89/122 2023-10-18T16:06:31.866 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 90/122 2023-10-18T16:06:31.866 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : xfsprogs-devel-5.0.0-8.el8.x86_64 91/122 2023-10-18T16:06:31.867 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 92/122 2023-10-18T16:06:31.867 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libuuid-devel-2.32.1-27.el8.x86_64 93/122 2023-10-18T16:06:31.867 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 94/122 2023-10-18T16:06:31.867 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 95/122 2023-10-18T16:06:31.868 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 96/122 2023-10-18T16:06:31.868 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : jq-1.5-12.el8.x86_64 97/122 2023-10-18T16:06:31.868 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : flex-2.6.1-9.el8.x86_64 98/122 2023-10-18T16:06:31.868 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 99/122 2023-10-18T16:06:31.869 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 100/122 2023-10-18T16:06:31.869 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 101/122 2023-10-18T16:06:31.869 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : bison-3.0.4-10.el8.x86_64 102/122 2023-10-18T16:06:31.869 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 103/122 2023-10-18T16:06:31.870 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 104/122 2023-10-18T16:06:31.870 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 105/122 2023-10-18T16:06:31.870 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 106/122 2023-10-18T16:06:31.870 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 107/122 2023-10-18T16:06:31.870 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 108/122 2023-10-18T16:06:31.871 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 109/122 2023-10-18T16:06:31.871 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 110/122 2023-10-18T16:06:31.871 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 111/122 2023-10-18T16:06:31.871 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 112/122 2023-10-18T16:06:31.871 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 113/122 2023-10-18T16:06:31.872 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 114/122 2023-10-18T16:06:31.872 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 115/122 2023-10-18T16:06:31.872 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 116/122 2023-10-18T16:06:31.872 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 117/122 2023-10-18T16:06:31.872 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : automake-1.16.1-7.el8.noarch 118/122 2023-10-18T16:06:31.872 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librados2-2:16.2.14-100.gb4ddd52c.el8.x86_64 119/122 2023-10-18T16:06:31.873 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 120/122 2023-10-18T16:06:31.873 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librbd1-2:16.2.14-100.gb4ddd52c.el8.x86_64 121/122 2023-10-18T16:06:32.503 INFO:teuthology.orchestra.run.smithi088.stdout:Installed products updated. 2023-10-18T16:06:32.647 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:06:32.647 INFO:teuthology.orchestra.run.smithi088.stdout:Upgraded: 2023-10-18T16:06:32.648 INFO:teuthology.orchestra.run.smithi088.stdout: librados2-2:16.2.14-100.gb4ddd52c.el8.x86_64 2023-10-18T16:06:32.648 INFO:teuthology.orchestra.run.smithi088.stdout: librbd1-2:16.2.14-100.gb4ddd52c.el8.x86_64 2023-10-18T16:06:32.648 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2023-10-18T16:06:32.648 INFO:teuthology.orchestra.run.smithi088.stdout: automake-1.16.1-7.el8.noarch 2023-10-18T16:06:32.649 INFO:teuthology.orchestra.run.smithi088.stdout: bison-3.0.4-10.el8.x86_64 2023-10-18T16:06:32.649 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-2:16.2.14-100.gb4ddd52c.el8.x86_64 2023-10-18T16:06:32.649 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-base-2:16.2.14-100.gb4ddd52c.el8.x86_64 2023-10-18T16:06:32.649 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-common-2:16.2.14-100.gb4ddd52c.el8.x86_64 2023-10-18T16:06:32.649 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-fuse-2:16.2.14-100.gb4ddd52c.el8.x86_64 2023-10-18T16:06:32.649 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-grafana-dashboards-2:16.2.14-100.gb4ddd52c.el8.noarch 2023-10-18T16:06:32.650 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-immutable-object-cache-2:16.2.14-100.gb4ddd52c.el8.x86_64 2023-10-18T16:06:32.650 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mds-2:16.2.14-100.gb4ddd52c.el8.x86_64 2023-10-18T16:06:32.650 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-2:16.2.14-100.gb4ddd52c.el8.x86_64 2023-10-18T16:06:32.650 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-cephadm-2:16.2.14-100.gb4ddd52c.el8.noarch 2023-10-18T16:06:32.650 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-dashboard-2:16.2.14-100.gb4ddd52c.el8.noarch 2023-10-18T16:06:32.651 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-diskprediction-local-2:16.2.14-100.gb4ddd52c.el8.noarch 2023-10-18T16:06:32.651 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-modules-core-2:16.2.14-100.gb4ddd52c.el8.noarch 2023-10-18T16:06:32.651 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-rook-2:16.2.14-100.gb4ddd52c.el8.noarch 2023-10-18T16:06:32.651 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mon-2:16.2.14-100.gb4ddd52c.el8.x86_64 2023-10-18T16:06:32.651 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-osd-2:16.2.14-100.gb4ddd52c.el8.x86_64 2023-10-18T16:06:32.651 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-prometheus-alerts-2:16.2.14-100.gb4ddd52c.el8.noarch 2023-10-18T16:06:32.652 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-radosgw-2:16.2.14-100.gb4ddd52c.el8.x86_64 2023-10-18T16:06:32.652 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-selinux-2:16.2.14-100.gb4ddd52c.el8.x86_64 2023-10-18T16:06:32.652 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-test-2:16.2.14-100.gb4ddd52c.el8.x86_64 2023-10-18T16:06:32.652 INFO:teuthology.orchestra.run.smithi088.stdout: cephadm-2:16.2.14-100.gb4ddd52c.el8.noarch 2023-10-18T16:06:32.652 INFO:teuthology.orchestra.run.smithi088.stdout: cephfs-mirror-2:16.2.14-100.gb4ddd52c.el8.x86_64 2023-10-18T16:06:32.653 INFO:teuthology.orchestra.run.smithi088.stdout: cephfs-top-2:16.2.14-100.gb4ddd52c.el8.noarch 2023-10-18T16:06:32.653 INFO:teuthology.orchestra.run.smithi088.stdout: elfutils-libelf-devel-0.182-3.el8.x86_64 2023-10-18T16:06:32.653 INFO:teuthology.orchestra.run.smithi088.stdout: flex-2.6.1-9.el8.x86_64 2023-10-18T16:06:32.653 INFO:teuthology.orchestra.run.smithi088.stdout: fuse-2.9.7-12.el8.x86_64 2023-10-18T16:06:32.653 INFO:teuthology.orchestra.run.smithi088.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-10-18T16:06:32.654 INFO:teuthology.orchestra.run.smithi088.stdout: jq-1.5-12.el8.x86_64 2023-10-18T16:06:32.654 INFO:teuthology.orchestra.run.smithi088.stdout: keyutils-libs-devel-1.5.10-6.el8.x86_64 2023-10-18T16:06:32.654 INFO:teuthology.orchestra.run.smithi088.stdout: krb5-devel-1.18.2-8.3.el8_4.x86_64 2023-10-18T16:06:32.654 INFO:teuthology.orchestra.run.smithi088.stdout: leveldb-1.22-1.el8.x86_64 2023-10-18T16:06:32.654 INFO:teuthology.orchestra.run.smithi088.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-10-18T16:06:32.654 INFO:teuthology.orchestra.run.smithi088.stdout: libcephfs-devel-2:16.2.14-100.gb4ddd52c.el8.x86_64 2023-10-18T16:06:32.655 INFO:teuthology.orchestra.run.smithi088.stdout: libcephfs2-2:16.2.14-100.gb4ddd52c.el8.x86_64 2023-10-18T16:06:32.655 INFO:teuthology.orchestra.run.smithi088.stdout: libcephsqlite-2:16.2.14-100.gb4ddd52c.el8.x86_64 2023-10-18T16:06:32.655 INFO:teuthology.orchestra.run.smithi088.stdout: libcom_err-devel-1.45.6-1.el8.x86_64 2023-10-18T16:06:32.656 INFO:teuthology.orchestra.run.smithi088.stdout: libgfortran-8.4.1-1.el8.x86_64 2023-10-18T16:06:32.656 INFO:teuthology.orchestra.run.smithi088.stdout: liboath-2.6.2-3.el8.x86_64 2023-10-18T16:06:32.656 INFO:teuthology.orchestra.run.smithi088.stdout: libquadmath-8.4.1-1.el8.x86_64 2023-10-18T16:06:32.656 INFO:teuthology.orchestra.run.smithi088.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-10-18T16:06:32.656 INFO:teuthology.orchestra.run.smithi088.stdout: librados-devel-2:16.2.14-100.gb4ddd52c.el8.x86_64 2023-10-18T16:06:32.657 INFO:teuthology.orchestra.run.smithi088.stdout: libradosstriper1-2:16.2.14-100.gb4ddd52c.el8.x86_64 2023-10-18T16:06:32.657 INFO:teuthology.orchestra.run.smithi088.stdout: librdkafka-0.11.4-1.el8.x86_64 2023-10-18T16:06:32.657 INFO:teuthology.orchestra.run.smithi088.stdout: librgw2-2:16.2.14-100.gb4ddd52c.el8.x86_64 2023-10-18T16:06:32.657 INFO:teuthology.orchestra.run.smithi088.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-10-18T16:06:32.657 INFO:teuthology.orchestra.run.smithi088.stdout: libsepol-devel-2.9-2.el8.x86_64 2023-10-18T16:06:32.658 INFO:teuthology.orchestra.run.smithi088.stdout: libtool-2.4.6-25.el8.x86_64 2023-10-18T16:06:32.658 INFO:teuthology.orchestra.run.smithi088.stdout: libunwind-1.3.1-3.el8.x86_64 2023-10-18T16:06:32.658 INFO:teuthology.orchestra.run.smithi088.stdout: libuuid-devel-2.32.1-27.el8.x86_64 2023-10-18T16:06:32.658 INFO:teuthology.orchestra.run.smithi088.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-10-18T16:06:32.658 INFO:teuthology.orchestra.run.smithi088.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-10-18T16:06:32.658 INFO:teuthology.orchestra.run.smithi088.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-10-18T16:06:32.659 INFO:teuthology.orchestra.run.smithi088.stdout: openblas-0.3.12-1.el8.x86_64 2023-10-18T16:06:32.659 INFO:teuthology.orchestra.run.smithi088.stdout: openblas-threads-0.3.12-1.el8.x86_64 2023-10-18T16:06:32.659 INFO:teuthology.orchestra.run.smithi088.stdout: openssl-devel-1:1.1.1g-15.el8_3.x86_64 2023-10-18T16:06:32.659 INFO:teuthology.orchestra.run.smithi088.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-10-18T16:06:32.660 INFO:teuthology.orchestra.run.smithi088.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-10-18T16:06:32.660 INFO:teuthology.orchestra.run.smithi088.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-10-18T16:06:32.660 INFO:teuthology.orchestra.run.smithi088.stdout: python3-apipkg-1.5-6.el8.noarch 2023-10-18T16:06:32.660 INFO:teuthology.orchestra.run.smithi088.stdout: python3-babel-2.5.1-5.el8.noarch 2023-10-18T16:06:32.661 INFO:teuthology.orchestra.run.smithi088.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-10-18T16:06:32.661 INFO:teuthology.orchestra.run.smithi088.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-10-18T16:06:32.661 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-10-18T16:06:32.661 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ceph-argparse-2:16.2.14-100.gb4ddd52c.el8.x86_64 2023-10-18T16:06:32.661 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ceph-common-2:16.2.14-100.gb4ddd52c.el8.x86_64 2023-10-18T16:06:32.662 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cephfs-2:16.2.14-100.gb4ddd52c.el8.x86_64 2023-10-18T16:06:32.662 INFO:teuthology.orchestra.run.smithi088.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-10-18T16:06:32.662 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-10-18T16:06:32.662 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-10-18T16:06:32.663 INFO:teuthology.orchestra.run.smithi088.stdout: python3-execnet-1.7.1-1.el8.noarch 2023-10-18T16:06:32.663 INFO:teuthology.orchestra.run.smithi088.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-10-18T16:06:32.663 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-6.2-6.el8.noarch 2023-10-18T16:06:32.663 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-10-18T16:06:32.664 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2023-10-18T16:06:32.664 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-10-18T16:06:32.664 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-10-18T16:06:32.664 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-10-18T16:06:32.665 INFO:teuthology.orchestra.run.smithi088.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-10-18T16:06:32.665 INFO:teuthology.orchestra.run.smithi088.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-10-18T16:06:32.665 INFO:teuthology.orchestra.run.smithi088.stdout: python3-mako-1.0.6-13.el8.noarch 2023-10-18T16:06:32.665 INFO:teuthology.orchestra.run.smithi088.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-10-18T16:06:32.665 INFO:teuthology.orchestra.run.smithi088.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-10-18T16:06:32.666 INFO:teuthology.orchestra.run.smithi088.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2023-10-18T16:06:32.666 INFO:teuthology.orchestra.run.smithi088.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2023-10-18T16:06:32.666 INFO:teuthology.orchestra.run.smithi088.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-10-18T16:06:32.667 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-10-18T16:06:32.667 INFO:teuthology.orchestra.run.smithi088.stdout: python3-portend-2.6-1.el8.noarch 2023-10-18T16:06:32.667 INFO:teuthology.orchestra.run.smithi088.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-10-18T16:06:32.667 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-10-18T16:06:32.667 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-10-18T16:06:32.668 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pytz-2017.2-9.el8.noarch 2023-10-18T16:06:32.668 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rados-2:16.2.14-100.gb4ddd52c.el8.x86_64 2023-10-18T16:06:32.668 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rbd-2:16.2.14-100.gb4ddd52c.el8.x86_64 2023-10-18T16:06:32.668 INFO:teuthology.orchestra.run.smithi088.stdout: python3-remoto-1.2.1-1.el8.noarch 2023-10-18T16:06:32.669 INFO:teuthology.orchestra.run.smithi088.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-10-18T16:06:32.669 INFO:teuthology.orchestra.run.smithi088.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-10-18T16:06:32.669 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rgw-2:16.2.14-100.gb4ddd52c.el8.x86_64 2023-10-18T16:06:32.669 INFO:teuthology.orchestra.run.smithi088.stdout: python3-routes-2.4.1-12.el8.noarch 2023-10-18T16:06:32.670 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rsa-4.9-2.el8.noarch 2023-10-18T16:06:32.670 INFO:teuthology.orchestra.run.smithi088.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2023-10-18T16:06:32.670 INFO:teuthology.orchestra.run.smithi088.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-10-18T16:06:32.671 INFO:teuthology.orchestra.run.smithi088.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-10-18T16:06:32.671 INFO:teuthology.orchestra.run.smithi088.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-10-18T16:06:32.671 INFO:teuthology.orchestra.run.smithi088.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-10-18T16:06:32.671 INFO:teuthology.orchestra.run.smithi088.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-10-18T16:06:32.671 INFO:teuthology.orchestra.run.smithi088.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-10-18T16:06:32.673 INFO:teuthology.orchestra.run.smithi088.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-10-18T16:06:32.673 INFO:teuthology.orchestra.run.smithi088.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-10-18T16:06:32.673 INFO:teuthology.orchestra.run.smithi088.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-10-18T16:06:32.674 INFO:teuthology.orchestra.run.smithi088.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-10-18T16:06:32.674 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-fuse-2:16.2.14-100.gb4ddd52c.el8.x86_64 2023-10-18T16:06:32.674 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-mirror-2:16.2.14-100.gb4ddd52c.el8.x86_64 2023-10-18T16:06:32.674 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-nbd-2:16.2.14-100.gb4ddd52c.el8.x86_64 2023-10-18T16:06:32.674 INFO:teuthology.orchestra.run.smithi088.stdout: socat-1.7.3.3-2.el8.x86_64 2023-10-18T16:06:32.675 INFO:teuthology.orchestra.run.smithi088.stdout: xfsprogs-devel-5.0.0-8.el8.x86_64 2023-10-18T16:06:32.675 INFO:teuthology.orchestra.run.smithi088.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-10-18T16:06:32.675 INFO:teuthology.orchestra.run.smithi088.stdout: zlib-devel-1.2.11-17.el8.x86_64 2023-10-18T16:06:32.676 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:06:32.676 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2023-10-18T16:06:32.920 DEBUG:teuthology.parallel:result is None 2023-10-18T16:06:34.006 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 122/122 2023-10-18T16:06:34.007 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:42 ago on Wed 18 Oct 2023 04:04:52 PM UTC. 2023-10-18T16:06:36.019 INFO:teuthology.orchestra.run.smithi151.stdout:Installed products updated. 2023-10-18T16:06:36.160 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-10-18T16:06:36.160 INFO:teuthology.orchestra.run.smithi151.stdout:Upgraded: 2023-10-18T16:06:36.160 INFO:teuthology.orchestra.run.smithi151.stdout: librados2-2:16.2.14-100.gb4ddd52c.el8.x86_64 2023-10-18T16:06:36.161 INFO:teuthology.orchestra.run.smithi151.stdout: librbd1-2:16.2.14-100.gb4ddd52c.el8.x86_64 2023-10-18T16:06:36.161 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2023-10-18T16:06:36.161 INFO:teuthology.orchestra.run.smithi151.stdout: automake-1.16.1-7.el8.noarch 2023-10-18T16:06:36.161 INFO:teuthology.orchestra.run.smithi151.stdout: bison-3.0.4-10.el8.x86_64 2023-10-18T16:06:36.162 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-2:16.2.14-100.gb4ddd52c.el8.x86_64 2023-10-18T16:06:36.162 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-base-2:16.2.14-100.gb4ddd52c.el8.x86_64 2023-10-18T16:06:36.162 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-common-2:16.2.14-100.gb4ddd52c.el8.x86_64 2023-10-18T16:06:36.162 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-fuse-2:16.2.14-100.gb4ddd52c.el8.x86_64 2023-10-18T16:06:36.162 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-grafana-dashboards-2:16.2.14-100.gb4ddd52c.el8.noarch 2023-10-18T16:06:36.163 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-immutable-object-cache-2:16.2.14-100.gb4ddd52c.el8.x86_64 2023-10-18T16:06:36.163 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mds-2:16.2.14-100.gb4ddd52c.el8.x86_64 2023-10-18T16:06:36.163 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-2:16.2.14-100.gb4ddd52c.el8.x86_64 2023-10-18T16:06:36.163 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-cephadm-2:16.2.14-100.gb4ddd52c.el8.noarch 2023-10-18T16:06:36.163 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-dashboard-2:16.2.14-100.gb4ddd52c.el8.noarch 2023-10-18T16:06:36.163 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-diskprediction-local-2:16.2.14-100.gb4ddd52c.el8.noarch 2023-10-18T16:06:36.164 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-modules-core-2:16.2.14-100.gb4ddd52c.el8.noarch 2023-10-18T16:06:36.164 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-rook-2:16.2.14-100.gb4ddd52c.el8.noarch 2023-10-18T16:06:36.164 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mon-2:16.2.14-100.gb4ddd52c.el8.x86_64 2023-10-18T16:06:36.164 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-osd-2:16.2.14-100.gb4ddd52c.el8.x86_64 2023-10-18T16:06:36.164 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-prometheus-alerts-2:16.2.14-100.gb4ddd52c.el8.noarch 2023-10-18T16:06:36.165 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-radosgw-2:16.2.14-100.gb4ddd52c.el8.x86_64 2023-10-18T16:06:36.165 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-selinux-2:16.2.14-100.gb4ddd52c.el8.x86_64 2023-10-18T16:06:36.165 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-test-2:16.2.14-100.gb4ddd52c.el8.x86_64 2023-10-18T16:06:36.165 INFO:teuthology.orchestra.run.smithi151.stdout: cephadm-2:16.2.14-100.gb4ddd52c.el8.noarch 2023-10-18T16:06:36.166 INFO:teuthology.orchestra.run.smithi151.stdout: cephfs-mirror-2:16.2.14-100.gb4ddd52c.el8.x86_64 2023-10-18T16:06:36.166 INFO:teuthology.orchestra.run.smithi151.stdout: cephfs-top-2:16.2.14-100.gb4ddd52c.el8.noarch 2023-10-18T16:06:36.166 INFO:teuthology.orchestra.run.smithi151.stdout: elfutils-libelf-devel-0.182-3.el8.x86_64 2023-10-18T16:06:36.166 INFO:teuthology.orchestra.run.smithi151.stdout: flex-2.6.1-9.el8.x86_64 2023-10-18T16:06:36.167 INFO:teuthology.orchestra.run.smithi151.stdout: fuse-2.9.7-12.el8.x86_64 2023-10-18T16:06:36.167 INFO:teuthology.orchestra.run.smithi151.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-10-18T16:06:36.167 INFO:teuthology.orchestra.run.smithi151.stdout: jq-1.5-12.el8.x86_64 2023-10-18T16:06:36.167 INFO:teuthology.orchestra.run.smithi151.stdout: keyutils-libs-devel-1.5.10-6.el8.x86_64 2023-10-18T16:06:36.167 INFO:teuthology.orchestra.run.smithi151.stdout: krb5-devel-1.18.2-8.3.el8_4.x86_64 2023-10-18T16:06:36.168 INFO:teuthology.orchestra.run.smithi151.stdout: leveldb-1.22-1.el8.x86_64 2023-10-18T16:06:36.168 INFO:teuthology.orchestra.run.smithi151.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-10-18T16:06:36.168 INFO:teuthology.orchestra.run.smithi151.stdout: libcephfs-devel-2:16.2.14-100.gb4ddd52c.el8.x86_64 2023-10-18T16:06:36.168 INFO:teuthology.orchestra.run.smithi151.stdout: libcephfs2-2:16.2.14-100.gb4ddd52c.el8.x86_64 2023-10-18T16:06:36.168 INFO:teuthology.orchestra.run.smithi151.stdout: libcephsqlite-2:16.2.14-100.gb4ddd52c.el8.x86_64 2023-10-18T16:06:36.169 INFO:teuthology.orchestra.run.smithi151.stdout: libcom_err-devel-1.45.6-1.el8.x86_64 2023-10-18T16:06:36.169 INFO:teuthology.orchestra.run.smithi151.stdout: libgfortran-8.4.1-1.el8.x86_64 2023-10-18T16:06:36.169 INFO:teuthology.orchestra.run.smithi151.stdout: liboath-2.6.2-3.el8.x86_64 2023-10-18T16:06:36.169 INFO:teuthology.orchestra.run.smithi151.stdout: libquadmath-8.4.1-1.el8.x86_64 2023-10-18T16:06:36.170 INFO:teuthology.orchestra.run.smithi151.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-10-18T16:06:36.170 INFO:teuthology.orchestra.run.smithi151.stdout: librados-devel-2:16.2.14-100.gb4ddd52c.el8.x86_64 2023-10-18T16:06:36.170 INFO:teuthology.orchestra.run.smithi151.stdout: libradosstriper1-2:16.2.14-100.gb4ddd52c.el8.x86_64 2023-10-18T16:06:36.170 INFO:teuthology.orchestra.run.smithi151.stdout: librdkafka-0.11.4-1.el8.x86_64 2023-10-18T16:06:36.170 INFO:teuthology.orchestra.run.smithi151.stdout: librgw2-2:16.2.14-100.gb4ddd52c.el8.x86_64 2023-10-18T16:06:36.171 INFO:teuthology.orchestra.run.smithi151.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-10-18T16:06:36.171 INFO:teuthology.orchestra.run.smithi151.stdout: libsepol-devel-2.9-2.el8.x86_64 2023-10-18T16:06:36.171 INFO:teuthology.orchestra.run.smithi151.stdout: libtool-2.4.6-25.el8.x86_64 2023-10-18T16:06:36.171 INFO:teuthology.orchestra.run.smithi151.stdout: libunwind-1.3.1-3.el8.x86_64 2023-10-18T16:06:36.171 INFO:teuthology.orchestra.run.smithi151.stdout: libuuid-devel-2.32.1-27.el8.x86_64 2023-10-18T16:06:36.172 INFO:teuthology.orchestra.run.smithi151.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-10-18T16:06:36.172 INFO:teuthology.orchestra.run.smithi151.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-10-18T16:06:36.172 INFO:teuthology.orchestra.run.smithi151.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-10-18T16:06:36.172 INFO:teuthology.orchestra.run.smithi151.stdout: openblas-0.3.12-1.el8.x86_64 2023-10-18T16:06:36.172 INFO:teuthology.orchestra.run.smithi151.stdout: openblas-threads-0.3.12-1.el8.x86_64 2023-10-18T16:06:36.173 INFO:teuthology.orchestra.run.smithi151.stdout: openssl-devel-1:1.1.1g-15.el8_3.x86_64 2023-10-18T16:06:36.173 INFO:teuthology.orchestra.run.smithi151.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-10-18T16:06:36.173 INFO:teuthology.orchestra.run.smithi151.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-10-18T16:06:36.173 INFO:teuthology.orchestra.run.smithi151.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-10-18T16:06:36.173 INFO:teuthology.orchestra.run.smithi151.stdout: python3-apipkg-1.5-6.el8.noarch 2023-10-18T16:06:36.174 INFO:teuthology.orchestra.run.smithi151.stdout: python3-babel-2.5.1-5.el8.noarch 2023-10-18T16:06:36.174 INFO:teuthology.orchestra.run.smithi151.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-10-18T16:06:36.174 INFO:teuthology.orchestra.run.smithi151.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-10-18T16:06:36.174 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-10-18T16:06:36.174 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ceph-argparse-2:16.2.14-100.gb4ddd52c.el8.x86_64 2023-10-18T16:06:36.174 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ceph-common-2:16.2.14-100.gb4ddd52c.el8.x86_64 2023-10-18T16:06:36.175 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cephfs-2:16.2.14-100.gb4ddd52c.el8.x86_64 2023-10-18T16:06:36.175 INFO:teuthology.orchestra.run.smithi151.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-10-18T16:06:36.175 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-10-18T16:06:36.175 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-10-18T16:06:36.175 INFO:teuthology.orchestra.run.smithi151.stdout: python3-execnet-1.7.1-1.el8.noarch 2023-10-18T16:06:36.176 INFO:teuthology.orchestra.run.smithi151.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-10-18T16:06:36.176 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-6.2-6.el8.noarch 2023-10-18T16:06:36.176 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-10-18T16:06:36.176 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2023-10-18T16:06:36.176 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-10-18T16:06:36.177 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-10-18T16:06:36.177 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-10-18T16:06:36.177 INFO:teuthology.orchestra.run.smithi151.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-10-18T16:06:36.177 INFO:teuthology.orchestra.run.smithi151.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-10-18T16:06:36.177 INFO:teuthology.orchestra.run.smithi151.stdout: python3-mako-1.0.6-13.el8.noarch 2023-10-18T16:06:36.178 INFO:teuthology.orchestra.run.smithi151.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-10-18T16:06:36.178 INFO:teuthology.orchestra.run.smithi151.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-10-18T16:06:36.178 INFO:teuthology.orchestra.run.smithi151.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2023-10-18T16:06:36.178 INFO:teuthology.orchestra.run.smithi151.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2023-10-18T16:06:36.178 INFO:teuthology.orchestra.run.smithi151.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-10-18T16:06:36.179 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-10-18T16:06:36.179 INFO:teuthology.orchestra.run.smithi151.stdout: python3-portend-2.6-1.el8.noarch 2023-10-18T16:06:36.179 INFO:teuthology.orchestra.run.smithi151.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-10-18T16:06:36.179 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-10-18T16:06:36.179 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-10-18T16:06:36.179 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pytz-2017.2-9.el8.noarch 2023-10-18T16:06:36.180 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rados-2:16.2.14-100.gb4ddd52c.el8.x86_64 2023-10-18T16:06:36.180 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rbd-2:16.2.14-100.gb4ddd52c.el8.x86_64 2023-10-18T16:06:36.181 INFO:teuthology.orchestra.run.smithi151.stdout: python3-remoto-1.2.1-1.el8.noarch 2023-10-18T16:06:36.181 INFO:teuthology.orchestra.run.smithi151.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-10-18T16:06:36.181 INFO:teuthology.orchestra.run.smithi151.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-10-18T16:06:36.181 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rgw-2:16.2.14-100.gb4ddd52c.el8.x86_64 2023-10-18T16:06:36.182 INFO:teuthology.orchestra.run.smithi151.stdout: python3-routes-2.4.1-12.el8.noarch 2023-10-18T16:06:36.182 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rsa-4.9-2.el8.noarch 2023-10-18T16:06:36.182 INFO:teuthology.orchestra.run.smithi151.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2023-10-18T16:06:36.182 INFO:teuthology.orchestra.run.smithi151.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-10-18T16:06:36.182 INFO:teuthology.orchestra.run.smithi151.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-10-18T16:06:36.183 INFO:teuthology.orchestra.run.smithi151.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-10-18T16:06:36.183 INFO:teuthology.orchestra.run.smithi151.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-10-18T16:06:36.183 INFO:teuthology.orchestra.run.smithi151.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-10-18T16:06:36.183 INFO:teuthology.orchestra.run.smithi151.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-10-18T16:06:36.183 INFO:teuthology.orchestra.run.smithi151.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-10-18T16:06:36.184 INFO:teuthology.orchestra.run.smithi151.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-10-18T16:06:36.184 INFO:teuthology.orchestra.run.smithi151.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-10-18T16:06:36.184 INFO:teuthology.orchestra.run.smithi151.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-10-18T16:06:36.184 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-fuse-2:16.2.14-100.gb4ddd52c.el8.x86_64 2023-10-18T16:06:36.184 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-mirror-2:16.2.14-100.gb4ddd52c.el8.x86_64 2023-10-18T16:06:36.184 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-nbd-2:16.2.14-100.gb4ddd52c.el8.x86_64 2023-10-18T16:06:36.185 INFO:teuthology.orchestra.run.smithi151.stdout: socat-1.7.3.3-2.el8.x86_64 2023-10-18T16:06:36.185 INFO:teuthology.orchestra.run.smithi151.stdout: xfsprogs-devel-5.0.0-8.el8.x86_64 2023-10-18T16:06:36.185 INFO:teuthology.orchestra.run.smithi151.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-10-18T16:06:36.186 INFO:teuthology.orchestra.run.smithi151.stdout: zlib-devel-1.2.11-17.el8.x86_64 2023-10-18T16:06:36.186 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-10-18T16:06:36.186 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2023-10-18T16:06:36.486 DEBUG:teuthology.parallel:result is None 2023-10-18T16:06:36.487 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5 2023-10-18T16:06:36.639 DEBUG:teuthology.orchestra.run.smithi088:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-18T16:06:36.663 INFO:teuthology.orchestra.run.smithi088.stdout:16.2.14-100.gb4ddd52c.el8 2023-10-18T16:06:36.663 INFO:teuthology.packaging:The installed version of ceph is 16.2.14-100.gb4ddd52c.el8 2023-10-18T16:06:36.664 INFO:teuthology.task.install:The correct ceph version 16.2.14-100.gb4ddd52c is installed. 2023-10-18T16:06:36.664 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5 2023-10-18T16:06:36.818 DEBUG:teuthology.orchestra.run.smithi151:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-18T16:06:36.838 INFO:teuthology.orchestra.run.smithi151.stdout:16.2.14-100.gb4ddd52c.el8 2023-10-18T16:06:36.838 INFO:teuthology.packaging:The installed version of ceph is 16.2.14-100.gb4ddd52c.el8 2023-10-18T16:06:36.839 INFO:teuthology.task.install:The correct ceph version 16.2.14-100.gb4ddd52c is installed. 2023-10-18T16:06:36.839 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-10-18T16:06:36.840 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-18T16:06:36.840 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-18T16:06:36.881 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-10-18T16:06:36.882 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-18T16:06:36.919 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-10-18T16:06:36.919 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-18T16:06:36.920 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/usr/bin/daemon-helper 2023-10-18T16:06:36.947 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-18T16:06:37.015 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-10-18T16:06:37.015 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/usr/bin/daemon-helper 2023-10-18T16:06:37.039 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-18T16:06:37.103 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-10-18T16:06:37.104 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-18T16:06:37.105 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-18T16:06:37.131 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-18T16:06:37.196 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-10-18T16:06:37.196 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-18T16:06:37.220 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-18T16:06:37.283 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-10-18T16:06:37.284 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-18T16:06:37.284 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/usr/bin/stdin-killer 2023-10-18T16:06:37.309 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-18T16:06:37.373 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-10-18T16:06:37.373 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/usr/bin/stdin-killer 2023-10-18T16:06:37.397 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-18T16:06:37.461 INFO:teuthology.run_tasks:Running task ceph... 2023-10-18T16:06:37.521 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-10-18T16:06:37.522 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 777 /var/log/ceph 2023-10-18T16:06:37.524 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod 777 /var/log/ceph 2023-10-18T16:06:37.548 INFO:tasks.ceph:Disabling ceph logrotate... 2023-10-18T16:06:37.548 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-18T16:06:37.589 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-18T16:06:37.612 INFO:tasks.ceph:Creating extra log directories... 2023-10-18T16:06:37.612 DEBUG:teuthology.orchestra.run.smithi088:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-18T16:06:37.652 DEBUG:teuthology.orchestra.run.smithi151:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-18T16:06:37.682 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2023-10-18T16:06:37.683 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2023-10-18T16:06:37.683 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2023-10-18T16:06:37.683 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-18T16:06:37.684 DEBUG:teuthology.orchestra.run.smithi088:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-10-18T16:06:37.728 DEBUG:teuthology.orchestra.run.smithi088:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-10-18T16:06:37.839 DEBUG:teuthology.orchestra.run.smithi088:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-10-18T16:06:37.863 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-10-18T16:06:37.864 DEBUG:teuthology.orchestra.run.smithi151:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-10-18T16:06:37.878 DEBUG:teuthology.orchestra.run.smithi151:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-10-18T16:06:37.988 DEBUG:teuthology.orchestra.run.smithi151:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-10-18T16:06:38.032 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-10-18T16:06:38.033 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': False, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_sleep_rank_change': 5000000.0, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'Replacing daemon mds'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-18T16:06:38.033 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/vshankar-2023-10-16_12:48:28-fs-wip-vshankar-testing-pacific-20231016.091648-testing-default-smithi/7429067', 'branch': 'wip-vshankar-testing-pacific-20231016.091648', 'description': 'fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}}', 'email': 'vshankar@redhat.com', 'first_in_suite': False, 'job_id': '7429067', 'kernel': {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': True, 'sha1': '47672119149b103ec7cec2cc65c063ccbb001ac7'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'vshankar-2023-10-16_12:48:28-fs-wip-vshankar-testing-pacific-20231016.091648-testing-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'rhel', 'os_version': '8.4', 'overrides': {'admin_socket': {'branch': 'wip-vshankar-testing-pacific-20231016.091648'}, 'ceph': {'cephfs': {'max_mds': 3, 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': False, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_sleep_rank_change': 5000000.0, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'Replacing daemon mds'], 'sha1': 'b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'check-counter': {'counters': {'mds': ['mds.exported', 'mds.imported']}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-vshankar-testing-pacific-20231016.091648', 'sha1': 'b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5'}}, 'owner': 'scheduled_vshankar@teuthology', 'priority': 80, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']], 'seed': 7825, 'sha1': 'b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5', 'sleep_before_teardown': 0, 'subset': '33/1024', 'suite': 'fs', 'suite_branch': 'wip-vshankar-testing-pacific-20231016.091648', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': 'b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5', 'targets': {'smithi088.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDM0J7RzHKtMvi+MMC04dICQhoKb8N07LvmP9S2NyG0Ljq9D1SSqj3y5PRJLxhMdmsMjbaiAzPfrTsQdrK9x9Ho=', 'smithi151.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBbNF+1REZY6QhkJY1gSEnsFbRSKC+civjX7dXcQZ0VlbSAYkufd5XtNVsJX3hiJyfcxEIIh87QC/wsq4npNqzM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': True, 'sha1': '47672119149b103ec7cec2cc65c063ccbb001ac7'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'flavor': 'default', 'sha1': 'b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5'}}, {'ceph': None}, {'kclient': None}, {'mds_thrash': None}, {'workunit': {'clients': {'all': ['fs/snaps']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '8cdab074dcca9a68965bc5a50e9c30b691949723', 'timestamp': '2023-10-16_12:48:28', 'tube': 'smithi', 'user': 'vshankar', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-10-18T16:06:38.034 DEBUG:teuthology.orchestra.run.smithi088:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-18T16:06:38.036 DEBUG:teuthology.orchestra.run.smithi151:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-18T16:06:38.081 DEBUG:teuthology.orchestra.run.smithi088:> sudo install -d -m0777 -- /var/run/ceph 2023-10-18T16:06:38.091 DEBUG:teuthology.orchestra.run.smithi151:> sudo install -d -m0777 -- /var/run/ceph 2023-10-18T16:06:38.144 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-18T16:06:38.145 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/scratch_devs of=/dev/stdout 2023-10-18T16:06:38.168 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-18T16:06:38.168 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_1 2023-10-18T16:06:38.223 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-18T16:06:38.224 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-18T16:06:38.224 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 6h/6d Inode: 218883 Links: 1 2023-10-18T16:06:38.224 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-18T16:06:38.225 INFO:teuthology.orchestra.run.smithi088.stdout:Context: system_u:object_r:device_t:s0 2023-10-18T16:06:38.225 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2023-10-18 16:06:27.949123118 +0000 2023-10-18T16:06:38.225 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2023-10-18 16:04:01.930401269 +0000 2023-10-18T16:06:38.225 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2023-10-18 16:04:01.930401269 +0000 2023-10-18T16:06:38.225 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: - 2023-10-18T16:06:38.226 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-18T16:06:38.287 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2023-10-18T16:06:38.288 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2023-10-18T16:06:38.288 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000272238 s, 1.9 MB/s 2023-10-18T16:06:38.289 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-18T16:06:38.345 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_2 2023-10-18T16:06:38.400 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-18T16:06:38.400 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-18T16:06:38.400 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 6h/6d Inode: 212758 Links: 1 2023-10-18T16:06:38.401 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-18T16:06:38.401 INFO:teuthology.orchestra.run.smithi088.stdout:Context: system_u:object_r:device_t:s0 2023-10-18T16:06:38.401 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2023-10-18 16:06:27.949123118 +0000 2023-10-18T16:06:38.401 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2023-10-18 16:04:02.429390658 +0000 2023-10-18T16:06:38.401 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2023-10-18 16:04:02.429390658 +0000 2023-10-18T16:06:38.402 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: - 2023-10-18T16:06:38.402 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-18T16:06:38.464 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2023-10-18T16:06:38.464 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2023-10-18T16:06:38.464 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000241949 s, 2.1 MB/s 2023-10-18T16:06:38.464 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-18T16:06:38.520 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_3 2023-10-18T16:06:38.574 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-18T16:06:38.575 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-18T16:06:38.575 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 6h/6d Inode: 216869 Links: 1 2023-10-18T16:06:38.575 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-18T16:06:38.575 INFO:teuthology.orchestra.run.smithi088.stdout:Context: system_u:object_r:device_t:s0 2023-10-18T16:06:38.576 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2023-10-18 16:06:27.949123118 +0000 2023-10-18T16:06:38.576 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2023-10-18 16:04:02.897380706 +0000 2023-10-18T16:06:38.576 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2023-10-18 16:04:02.897380706 +0000 2023-10-18T16:06:38.576 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: - 2023-10-18T16:06:38.577 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-18T16:06:38.637 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2023-10-18T16:06:38.637 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2023-10-18T16:06:38.638 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000204817 s, 2.5 MB/s 2023-10-18T16:06:38.639 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-18T16:06:38.695 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_4 2023-10-18T16:06:38.750 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-18T16:06:38.750 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-18T16:06:38.750 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 6h/6d Inode: 216998 Links: 1 2023-10-18T16:06:38.751 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-18T16:06:38.751 INFO:teuthology.orchestra.run.smithi088.stdout:Context: system_u:object_r:device_t:s0 2023-10-18T16:06:38.751 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2023-10-18 16:06:27.949123118 +0000 2023-10-18T16:06:38.751 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2023-10-18 16:04:03.390370222 +0000 2023-10-18T16:06:38.751 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2023-10-18 16:04:03.390370222 +0000 2023-10-18T16:06:38.752 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: - 2023-10-18T16:06:38.752 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-18T16:06:38.813 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2023-10-18T16:06:38.813 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2023-10-18T16:06:38.813 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000192763 s, 2.7 MB/s 2023-10-18T16:06:38.814 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-18T16:06:38.870 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-18T16:06:38.870 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-10-18T16:06:38.870 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/scratch_devs of=/dev/stdout 2023-10-18T16:06:38.886 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-18T16:06:38.886 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_1 2023-10-18T16:06:38.942 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-18T16:06:38.942 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-18T16:06:38.942 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 6h/6d Inode: 221819 Links: 1 2023-10-18T16:06:38.943 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-18T16:06:38.943 INFO:teuthology.orchestra.run.smithi151.stdout:Context: system_u:object_r:device_t:s0 2023-10-18T16:06:38.943 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2023-10-18 16:06:31.389526557 +0000 2023-10-18T16:06:38.943 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2023-10-18 16:04:05.915941598 +0000 2023-10-18T16:06:38.943 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2023-10-18 16:04:05.915941598 +0000 2023-10-18T16:06:38.944 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: - 2023-10-18T16:06:38.944 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-18T16:06:39.006 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2023-10-18T16:06:39.007 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2023-10-18T16:06:39.007 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000348944 s, 1.5 MB/s 2023-10-18T16:06:39.008 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-18T16:06:39.065 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_2 2023-10-18T16:06:39.119 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-18T16:06:39.119 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-18T16:06:39.120 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 6h/6d Inode: 222938 Links: 1 2023-10-18T16:06:39.120 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-18T16:06:39.120 INFO:teuthology.orchestra.run.smithi151.stdout:Context: system_u:object_r:device_t:s0 2023-10-18T16:06:39.120 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2023-10-18 16:06:31.389526557 +0000 2023-10-18T16:06:39.120 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2023-10-18 16:04:06.403935700 +0000 2023-10-18T16:06:39.121 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2023-10-18 16:04:06.403935700 +0000 2023-10-18T16:06:39.121 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: - 2023-10-18T16:06:39.121 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-18T16:06:39.182 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2023-10-18T16:06:39.182 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2023-10-18T16:06:39.183 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000199566 s, 2.6 MB/s 2023-10-18T16:06:39.184 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-18T16:06:39.238 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_3 2023-10-18T16:06:39.293 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-18T16:06:39.293 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-18T16:06:39.293 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 6h/6d Inode: 221005 Links: 1 2023-10-18T16:06:39.293 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-18T16:06:39.294 INFO:teuthology.orchestra.run.smithi151.stdout:Context: system_u:object_r:device_t:s0 2023-10-18T16:06:39.294 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2023-10-18 16:06:31.389526557 +0000 2023-10-18T16:06:39.294 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2023-10-18 16:04:06.895929753 +0000 2023-10-18T16:06:39.294 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2023-10-18 16:04:06.895929753 +0000 2023-10-18T16:06:39.295 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: - 2023-10-18T16:06:39.295 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-18T16:06:39.355 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2023-10-18T16:06:39.355 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2023-10-18T16:06:39.355 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000237691 s, 2.2 MB/s 2023-10-18T16:06:39.356 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-18T16:06:39.411 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_4 2023-10-18T16:06:39.466 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-18T16:06:39.467 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-18T16:06:39.467 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 6h/6d Inode: 221103 Links: 1 2023-10-18T16:06:39.467 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-18T16:06:39.467 INFO:teuthology.orchestra.run.smithi151.stdout:Context: system_u:object_r:device_t:s0 2023-10-18T16:06:39.468 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2023-10-18 16:06:31.389526557 +0000 2023-10-18T16:06:39.468 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2023-10-18 16:04:07.408923553 +0000 2023-10-18T16:06:39.468 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2023-10-18 16:04:07.408923553 +0000 2023-10-18T16:06:39.468 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: - 2023-10-18T16:06:39.468 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-18T16:06:39.528 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2023-10-18T16:06:39.528 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2023-10-18T16:06:39.529 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000181979 s, 2.8 MB/s 2023-10-18T16:06:39.530 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-18T16:06:39.584 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-18T16:06:39.585 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi088.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi151.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-10-18T16:06:39.585 INFO:tasks.ceph:Generating config... 2023-10-18T16:06:39.586 INFO:tasks.ceph:[client] client mount timeout = 600 2023-10-18T16:06:39.586 INFO:tasks.ceph:[client] debug client = 20 2023-10-18T16:06:39.587 INFO:tasks.ceph:[client] debug ms = 1 2023-10-18T16:06:39.587 INFO:tasks.ceph:[client] rados mon op timeout = 900 2023-10-18T16:06:39.587 INFO:tasks.ceph:[client] rados osd op timeout = 900 2023-10-18T16:06:39.588 INFO:tasks.ceph:[global] ms die on skipped message = False 2023-10-18T16:06:39.588 INFO:tasks.ceph:[mds] debug mds = 20 2023-10-18T16:06:39.588 INFO:tasks.ceph:[mds] debug ms = 1 2023-10-18T16:06:39.588 INFO:tasks.ceph:[mds] mds bal fragment size max = 10000 2023-10-18T16:06:39.589 INFO:tasks.ceph:[mds] mds bal merge size = 5 2023-10-18T16:06:39.589 INFO:tasks.ceph:[mds] mds bal split bits = 3 2023-10-18T16:06:39.589 INFO:tasks.ceph:[mds] mds bal split size = 100 2023-10-18T16:06:39.589 INFO:tasks.ceph:[mds] mds debug frag = True 2023-10-18T16:06:39.590 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2023-10-18T16:06:39.590 INFO:tasks.ceph:[mds] mds oft prefetch dirfrags = False 2023-10-18T16:06:39.590 INFO:tasks.ceph:[mds] mds op complaint time = 180 2023-10-18T16:06:39.590 INFO:tasks.ceph:[mds] mds verify scatter = True 2023-10-18T16:06:39.591 INFO:tasks.ceph:[mds] mds_sleep_rank_change = 5000000.0 2023-10-18T16:06:39.591 INFO:tasks.ceph:[mds] osd op complaint time = 180 2023-10-18T16:06:39.591 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2023-10-18T16:06:39.591 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2023-10-18T16:06:39.592 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-18T16:06:39.592 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-18T16:06:39.592 INFO:tasks.ceph:[mon] debug mon = 20 2023-10-18T16:06:39.593 INFO:tasks.ceph:[mon] debug ms = 1 2023-10-18T16:06:39.593 INFO:tasks.ceph:[mon] debug paxos = 20 2023-10-18T16:06:39.593 INFO:tasks.ceph:[mon] mon op complaint time = 120 2023-10-18T16:06:39.593 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-10-18T16:06:39.594 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2023-10-18T16:06:39.594 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-10-18T16:06:39.594 INFO:tasks.ceph:[osd] debug bluefs = 20 2023-10-18T16:06:39.594 INFO:tasks.ceph:[osd] debug bluestore = 20 2023-10-18T16:06:39.595 INFO:tasks.ceph:[osd] debug ms = 1 2023-10-18T16:06:39.595 INFO:tasks.ceph:[osd] debug osd = 20 2023-10-18T16:06:39.595 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-10-18T16:06:39.595 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-10-18T16:06:39.596 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-10-18T16:06:39.596 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-10-18T16:06:39.596 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-10-18T16:06:39.596 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2023-10-18T16:06:39.597 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-10-18T16:06:39.597 INFO:tasks.ceph:[osd] osd op complaint time = 180 2023-10-18T16:06:39.597 INFO:tasks.ceph:Setting up mon.a... 2023-10-18T16:06:39.598 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-10-18T16:06:39.670 INFO:teuthology.orchestra.run.smithi088.stdout:creating /etc/ceph/ceph.keyring 2023-10-18T16:06:39.672 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-10-18T16:06:39.713 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-18T16:06:39.779 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.88'), ('mon.b', '172.21.15.151'), ('mon.c', '[v2:172.21.15.151:3301,v1:172.21.15.151:6790]')] 2023-10-18T16:06:39.779 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.88,172.21.15.151,[v2:172.21.15.151:3301,v1:172.21.15.151:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.e': {}, 'mds.b': {}, 'mds.d': {}, 'mds.f': {}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': False, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_sleep_rank_change': 5000000.0, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2023-10-18T16:06:39.780 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-18T16:06:39.781 DEBUG:teuthology.orchestra.run.smithi088:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-18T16:06:39.834 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.88 --add b 172.21.15.151 --addv c '[v2:172.21.15.151:3301,v1:172.21.15.151:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2023-10-18T16:06:39.915 INFO:teuthology.orchestra.run.smithi088.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-10-18T16:06:39.915 INFO:teuthology.orchestra.run.smithi088.stdout:monmaptool: generated fsid fb1f77e8-ead0-4f90-8128-fdf5137eb16f 2023-10-18T16:06:39.915 INFO:teuthology.orchestra.run.smithi088.stdout:epoch 0 2023-10-18T16:06:39.916 INFO:teuthology.orchestra.run.smithi088.stdout:fsid fb1f77e8-ead0-4f90-8128-fdf5137eb16f 2023-10-18T16:06:39.916 INFO:teuthology.orchestra.run.smithi088.stdout:last_changed 2023-10-18T16:06:39.912936+0000 2023-10-18T16:06:39.916 INFO:teuthology.orchestra.run.smithi088.stdout:created 2023-10-18T16:06:39.912936+0000 2023-10-18T16:06:39.916 INFO:teuthology.orchestra.run.smithi088.stdout:min_mon_release 0 (unknown) 2023-10-18T16:06:39.917 INFO:teuthology.orchestra.run.smithi088.stdout:election_strategy: 1 2023-10-18T16:06:39.917 INFO:teuthology.orchestra.run.smithi088.stdout:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-10-18T16:06:39.917 INFO:teuthology.orchestra.run.smithi088.stdout:1: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.b 2023-10-18T16:06:39.917 INFO:teuthology.orchestra.run.smithi088.stdout:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-10-18T16:06:39.917 INFO:teuthology.orchestra.run.smithi088.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-10-18T16:06:39.919 DEBUG:teuthology.orchestra.run.smithi088:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-18T16:06:39.973 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID fb1f77e8-ead0-4f90-8128-fdf5137eb16f... 2023-10-18T16:06:39.974 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-18T16:06:40.016 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-18T16:06:40.055 INFO:teuthology.orchestra.run.smithi088.stdout:[global] 2023-10-18T16:06:40.056 INFO:teuthology.orchestra.run.smithi088.stdout: chdir = "" 2023-10-18T16:06:40.056 INFO:teuthology.orchestra.run.smithi088.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-18T16:06:40.056 INFO:teuthology.orchestra.run.smithi088.stdout: auth supported = cephx 2023-10-18T16:06:40.056 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:06:40.056 INFO:teuthology.orchestra.run.smithi088.stdout: filestore xattr use omap = true 2023-10-18T16:06:40.057 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:06:40.057 INFO:teuthology.orchestra.run.smithi088.stdout: mon clock drift allowed = 1.000 2023-10-18T16:06:40.057 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:06:40.057 INFO:teuthology.orchestra.run.smithi088.stdout: osd crush chooseleaf type = 0 2023-10-18T16:06:40.057 INFO:teuthology.orchestra.run.smithi088.stdout: auth debug = true 2023-10-18T16:06:40.058 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:06:40.058 INFO:teuthology.orchestra.run.smithi088.stdout: ms die on old message = true 2023-10-18T16:06:40.058 INFO:teuthology.orchestra.run.smithi088.stdout: ms die on bug = true 2023-10-18T16:06:40.058 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:06:40.058 INFO:teuthology.orchestra.run.smithi088.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-18T16:06:40.058 INFO:teuthology.orchestra.run.smithi088.stdout: mon pg warn max object skew = 0 2023-10-18T16:06:40.059 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:06:40.059 INFO:teuthology.orchestra.run.smithi088.stdout: # disable pg_autoscaler by default for new pools 2023-10-18T16:06:40.059 INFO:teuthology.orchestra.run.smithi088.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-18T16:06:40.059 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:06:40.059 INFO:teuthology.orchestra.run.smithi088.stdout: osd pool default size = 2 2023-10-18T16:06:40.060 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:06:40.060 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd allow primary affinity = true 2023-10-18T16:06:40.060 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd allow pg remap = true 2023-10-18T16:06:40.060 INFO:teuthology.orchestra.run.smithi088.stdout: mon warn on legacy crush tunables = false 2023-10-18T16:06:40.060 INFO:teuthology.orchestra.run.smithi088.stdout: mon warn on crush straw calc version zero = false 2023-10-18T16:06:40.060 INFO:teuthology.orchestra.run.smithi088.stdout: mon warn on no sortbitwise = false 2023-10-18T16:06:40.061 INFO:teuthology.orchestra.run.smithi088.stdout: mon warn on osd down out interval zero = false 2023-10-18T16:06:40.061 INFO:teuthology.orchestra.run.smithi088.stdout: mon warn on too few osds = false 2023-10-18T16:06:40.061 INFO:teuthology.orchestra.run.smithi088.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-18T16:06:40.061 INFO:teuthology.orchestra.run.smithi088.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-18T16:06:40.061 INFO:teuthology.orchestra.run.smithi088.stdout: mon_allow_pool_size_one = true 2023-10-18T16:06:40.062 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:06:40.062 INFO:teuthology.orchestra.run.smithi088.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2023-10-18T16:06:40.062 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:06:40.062 INFO:teuthology.orchestra.run.smithi088.stdout: osd default data pool replay window = 5 2023-10-18T16:06:40.062 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:06:40.063 INFO:teuthology.orchestra.run.smithi088.stdout: mon allow pool delete = true 2023-10-18T16:06:40.063 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:06:40.063 INFO:teuthology.orchestra.run.smithi088.stdout: mon cluster log file level = debug 2023-10-18T16:06:40.063 INFO:teuthology.orchestra.run.smithi088.stdout: debug asserts on shutdown = true 2023-10-18T16:06:40.063 INFO:teuthology.orchestra.run.smithi088.stdout: mon health detail to clog = false 2023-10-18T16:06:40.063 INFO:teuthology.orchestra.run.smithi088.stdout: mon host = "172.21.15.88,172.21.15.151,[v2:172.21.15.151:3301,v1:172.21.15.151:6790]" 2023-10-18T16:06:40.064 INFO:teuthology.orchestra.run.smithi088.stdout: ms die on skipped message = False 2023-10-18T16:06:40.064 INFO:teuthology.orchestra.run.smithi088.stdout: fsid = fb1f77e8-ead0-4f90-8128-fdf5137eb16f 2023-10-18T16:06:40.064 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:06:40.064 INFO:teuthology.orchestra.run.smithi088.stdout:[osd] 2023-10-18T16:06:40.064 INFO:teuthology.orchestra.run.smithi088.stdout: osd journal size = 100 2023-10-18T16:06:40.065 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:06:40.065 INFO:teuthology.orchestra.run.smithi088.stdout: osd scrub load threshold = 5.0 2023-10-18T16:06:40.065 INFO:teuthology.orchestra.run.smithi088.stdout: osd scrub max interval = 600 2023-10-18T16:06:40.065 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:06:40.065 INFO:teuthology.orchestra.run.smithi088.stdout: osd recover clone overlap = true 2023-10-18T16:06:40.065 INFO:teuthology.orchestra.run.smithi088.stdout: osd recovery max chunk = 1048576 2023-10-18T16:06:40.066 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:06:40.066 INFO:teuthology.orchestra.run.smithi088.stdout: osd debug shutdown = true 2023-10-18T16:06:40.066 INFO:teuthology.orchestra.run.smithi088.stdout: osd debug op order = true 2023-10-18T16:06:40.066 INFO:teuthology.orchestra.run.smithi088.stdout: osd debug verify stray on activate = true 2023-10-18T16:06:40.066 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:06:40.067 INFO:teuthology.orchestra.run.smithi088.stdout: osd open classes on start = true 2023-10-18T16:06:40.067 INFO:teuthology.orchestra.run.smithi088.stdout: osd debug pg log writeout = true 2023-10-18T16:06:40.067 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:06:40.067 INFO:teuthology.orchestra.run.smithi088.stdout: osd deep scrub update digest min age = 30 2023-10-18T16:06:40.067 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:06:40.067 INFO:teuthology.orchestra.run.smithi088.stdout: osd map max advance = 10 2023-10-18T16:06:40.068 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:06:40.068 INFO:teuthology.orchestra.run.smithi088.stdout: journal zero on create = true 2023-10-18T16:06:40.068 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:06:40.068 INFO:teuthology.orchestra.run.smithi088.stdout: filestore ondisk finisher threads = 3 2023-10-18T16:06:40.068 INFO:teuthology.orchestra.run.smithi088.stdout: filestore apply finisher threads = 3 2023-10-18T16:06:40.069 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:06:40.069 INFO:teuthology.orchestra.run.smithi088.stdout: bdev debug aio = true 2023-10-18T16:06:40.069 INFO:teuthology.orchestra.run.smithi088.stdout: osd debug misdirected ops = true 2023-10-18T16:06:40.069 INFO:teuthology.orchestra.run.smithi088.stdout: bluestore block size = 96636764160 2023-10-18T16:06:40.069 INFO:teuthology.orchestra.run.smithi088.stdout: bluestore compression mode = aggressive 2023-10-18T16:06:40.069 INFO:teuthology.orchestra.run.smithi088.stdout: bluestore fsck on mount = True 2023-10-18T16:06:40.070 INFO:teuthology.orchestra.run.smithi088.stdout: debug bluefs = 20 2023-10-18T16:06:40.070 INFO:teuthology.orchestra.run.smithi088.stdout: debug bluestore = 20 2023-10-18T16:06:40.070 INFO:teuthology.orchestra.run.smithi088.stdout: debug ms = 1 2023-10-18T16:06:40.070 INFO:teuthology.orchestra.run.smithi088.stdout: debug osd = 20 2023-10-18T16:06:40.070 INFO:teuthology.orchestra.run.smithi088.stdout: debug rocksdb = 10 2023-10-18T16:06:40.071 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd backfillfull_ratio = 0.85 2023-10-18T16:06:40.071 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd full ratio = 0.9 2023-10-18T16:06:40.071 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd nearfull ratio = 0.8 2023-10-18T16:06:40.071 INFO:teuthology.orchestra.run.smithi088.stdout: osd failsafe full ratio = 0.95 2023-10-18T16:06:40.071 INFO:teuthology.orchestra.run.smithi088.stdout: osd max markdown count = 1000 2023-10-18T16:06:40.071 INFO:teuthology.orchestra.run.smithi088.stdout: osd objectstore = bluestore 2023-10-18T16:06:40.072 INFO:teuthology.orchestra.run.smithi088.stdout: osd op complaint time = 180 2023-10-18T16:06:40.072 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:06:40.072 INFO:teuthology.orchestra.run.smithi088.stdout:[mgr] 2023-10-18T16:06:40.072 INFO:teuthology.orchestra.run.smithi088.stdout: debug ms = 1 2023-10-18T16:06:40.072 INFO:teuthology.orchestra.run.smithi088.stdout: debug mgr = 20 2023-10-18T16:06:40.072 INFO:teuthology.orchestra.run.smithi088.stdout: debug mon = 20 2023-10-18T16:06:40.073 INFO:teuthology.orchestra.run.smithi088.stdout: debug auth = 20 2023-10-18T16:06:40.073 INFO:teuthology.orchestra.run.smithi088.stdout: mon reweight min pgs per osd = 4 2023-10-18T16:06:40.073 INFO:teuthology.orchestra.run.smithi088.stdout: mon reweight min bytes per osd = 10 2023-10-18T16:06:40.073 INFO:teuthology.orchestra.run.smithi088.stdout: mgr/telemetry/nag = false 2023-10-18T16:06:40.073 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:06:40.073 INFO:teuthology.orchestra.run.smithi088.stdout:[mon] 2023-10-18T16:06:40.074 INFO:teuthology.orchestra.run.smithi088.stdout: debug ms = 1 2023-10-18T16:06:40.074 INFO:teuthology.orchestra.run.smithi088.stdout: debug mon = 20 2023-10-18T16:06:40.074 INFO:teuthology.orchestra.run.smithi088.stdout: debug paxos = 20 2023-10-18T16:06:40.074 INFO:teuthology.orchestra.run.smithi088.stdout: debug auth = 20 2023-10-18T16:06:40.074 INFO:teuthology.orchestra.run.smithi088.stdout: mon data avail warn = 5 2023-10-18T16:06:40.075 INFO:teuthology.orchestra.run.smithi088.stdout: mon mgr mkfs grace = 240 2023-10-18T16:06:40.075 INFO:teuthology.orchestra.run.smithi088.stdout: mon reweight min pgs per osd = 4 2023-10-18T16:06:40.075 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd reporter subtree level = osd 2023-10-18T16:06:40.075 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd prime pg temp = true 2023-10-18T16:06:40.075 INFO:teuthology.orchestra.run.smithi088.stdout: mon reweight min bytes per osd = 10 2023-10-18T16:06:40.076 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:06:40.076 INFO:teuthology.orchestra.run.smithi088.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-18T16:06:40.076 INFO:teuthology.orchestra.run.smithi088.stdout: auth mon ticket ttl = 660 # 11m 2023-10-18T16:06:40.076 INFO:teuthology.orchestra.run.smithi088.stdout: auth service ticket ttl = 240 # 4m 2023-10-18T16:06:40.076 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:06:40.076 INFO:teuthology.orchestra.run.smithi088.stdout: # don't complain about insecure global_id in the test suite 2023-10-18T16:06:40.077 INFO:teuthology.orchestra.run.smithi088.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-18T16:06:40.077 INFO:teuthology.orchestra.run.smithi088.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-18T16:06:40.077 INFO:teuthology.orchestra.run.smithi088.stdout: mon op complaint time = 120 2023-10-18T16:06:40.077 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:06:40.077 INFO:teuthology.orchestra.run.smithi088.stdout:[client] 2023-10-18T16:06:40.077 INFO:teuthology.orchestra.run.smithi088.stdout: rgw cache enabled = true 2023-10-18T16:06:40.078 INFO:teuthology.orchestra.run.smithi088.stdout: rgw enable ops log = true 2023-10-18T16:06:40.078 INFO:teuthology.orchestra.run.smithi088.stdout: rgw enable usage log = true 2023-10-18T16:06:40.078 INFO:teuthology.orchestra.run.smithi088.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-18T16:06:40.078 INFO:teuthology.orchestra.run.smithi088.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-18T16:06:40.078 INFO:teuthology.orchestra.run.smithi088.stdout: client mount timeout = 600 2023-10-18T16:06:40.079 INFO:teuthology.orchestra.run.smithi088.stdout: debug client = 20 2023-10-18T16:06:40.079 INFO:teuthology.orchestra.run.smithi088.stdout: debug ms = 1 2023-10-18T16:06:40.079 INFO:teuthology.orchestra.run.smithi088.stdout: rados mon op timeout = 900 2023-10-18T16:06:40.079 INFO:teuthology.orchestra.run.smithi088.stdout: rados osd op timeout = 900 2023-10-18T16:06:40.079 INFO:teuthology.orchestra.run.smithi088.stdout:[mon.a] 2023-10-18T16:06:40.079 INFO:teuthology.orchestra.run.smithi088.stdout:[mon.b] 2023-10-18T16:06:40.080 INFO:teuthology.orchestra.run.smithi088.stdout:[mon.c] 2023-10-18T16:06:40.080 INFO:teuthology.orchestra.run.smithi088.stdout:[mds.a] 2023-10-18T16:06:40.080 INFO:teuthology.orchestra.run.smithi088.stdout:[mds.c] 2023-10-18T16:06:40.080 INFO:teuthology.orchestra.run.smithi088.stdout:[mds.e] 2023-10-18T16:06:40.080 INFO:teuthology.orchestra.run.smithi088.stdout:[mds.b] 2023-10-18T16:06:40.081 INFO:teuthology.orchestra.run.smithi088.stdout:[mds.d] 2023-10-18T16:06:40.081 INFO:teuthology.orchestra.run.smithi088.stdout:[mds.f] 2023-10-18T16:06:40.081 INFO:teuthology.orchestra.run.smithi088.stdout:[mds] 2023-10-18T16:06:40.081 INFO:teuthology.orchestra.run.smithi088.stdout: debug mds = 20 2023-10-18T16:06:40.081 INFO:teuthology.orchestra.run.smithi088.stdout: debug ms = 1 2023-10-18T16:06:40.081 INFO:teuthology.orchestra.run.smithi088.stdout: mds bal fragment size max = 10000 2023-10-18T16:06:40.082 INFO:teuthology.orchestra.run.smithi088.stdout: mds bal merge size = 5 2023-10-18T16:06:40.082 INFO:teuthology.orchestra.run.smithi088.stdout: mds bal split bits = 3 2023-10-18T16:06:40.082 INFO:teuthology.orchestra.run.smithi088.stdout: mds bal split size = 100 2023-10-18T16:06:40.082 INFO:teuthology.orchestra.run.smithi088.stdout: mds debug frag = True 2023-10-18T16:06:40.082 INFO:teuthology.orchestra.run.smithi088.stdout: mds debug scatterstat = True 2023-10-18T16:06:40.083 INFO:teuthology.orchestra.run.smithi088.stdout: mds oft prefetch dirfrags = False 2023-10-18T16:06:40.083 INFO:teuthology.orchestra.run.smithi088.stdout: mds op complaint time = 180 2023-10-18T16:06:40.083 INFO:teuthology.orchestra.run.smithi088.stdout: mds verify scatter = True 2023-10-18T16:06:40.083 INFO:teuthology.orchestra.run.smithi088.stdout: mds_sleep_rank_change = 5000000.0 2023-10-18T16:06:40.083 INFO:teuthology.orchestra.run.smithi088.stdout: osd op complaint time = 180 2023-10-18T16:06:40.083 INFO:teuthology.orchestra.run.smithi088.stdout: rados mon op timeout = 900 2023-10-18T16:06:40.084 INFO:teuthology.orchestra.run.smithi088.stdout: rados osd op timeout = 900 2023-10-18T16:06:40.085 INFO:teuthology.orchestra.run.smithi151.stdout:[global] 2023-10-18T16:06:40.085 INFO:teuthology.orchestra.run.smithi151.stdout: chdir = "" 2023-10-18T16:06:40.085 INFO:teuthology.orchestra.run.smithi151.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-18T16:06:40.086 INFO:teuthology.orchestra.run.smithi151.stdout: auth supported = cephx 2023-10-18T16:06:40.086 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-10-18T16:06:40.086 INFO:teuthology.orchestra.run.smithi151.stdout: filestore xattr use omap = true 2023-10-18T16:06:40.086 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-10-18T16:06:40.086 INFO:teuthology.orchestra.run.smithi151.stdout: mon clock drift allowed = 1.000 2023-10-18T16:06:40.087 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-10-18T16:06:40.087 INFO:teuthology.orchestra.run.smithi151.stdout: osd crush chooseleaf type = 0 2023-10-18T16:06:40.087 INFO:teuthology.orchestra.run.smithi151.stdout: auth debug = true 2023-10-18T16:06:40.087 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-10-18T16:06:40.087 INFO:teuthology.orchestra.run.smithi151.stdout: ms die on old message = true 2023-10-18T16:06:40.088 INFO:teuthology.orchestra.run.smithi151.stdout: ms die on bug = true 2023-10-18T16:06:40.088 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-10-18T16:06:40.088 INFO:teuthology.orchestra.run.smithi151.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-18T16:06:40.088 INFO:teuthology.orchestra.run.smithi151.stdout: mon pg warn max object skew = 0 2023-10-18T16:06:40.088 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-10-18T16:06:40.089 INFO:teuthology.orchestra.run.smithi151.stdout: # disable pg_autoscaler by default for new pools 2023-10-18T16:06:40.089 INFO:teuthology.orchestra.run.smithi151.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-18T16:06:40.089 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-10-18T16:06:40.089 INFO:teuthology.orchestra.run.smithi151.stdout: osd pool default size = 2 2023-10-18T16:06:40.089 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-10-18T16:06:40.089 INFO:teuthology.orchestra.run.smithi151.stdout: mon osd allow primary affinity = true 2023-10-18T16:06:40.090 INFO:teuthology.orchestra.run.smithi151.stdout: mon osd allow pg remap = true 2023-10-18T16:06:40.090 INFO:teuthology.orchestra.run.smithi151.stdout: mon warn on legacy crush tunables = false 2023-10-18T16:06:40.090 INFO:teuthology.orchestra.run.smithi151.stdout: mon warn on crush straw calc version zero = false 2023-10-18T16:06:40.090 INFO:teuthology.orchestra.run.smithi151.stdout: mon warn on no sortbitwise = false 2023-10-18T16:06:40.090 INFO:teuthology.orchestra.run.smithi151.stdout: mon warn on osd down out interval zero = false 2023-10-18T16:06:40.091 INFO:teuthology.orchestra.run.smithi151.stdout: mon warn on too few osds = false 2023-10-18T16:06:40.091 INFO:teuthology.orchestra.run.smithi151.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-18T16:06:40.091 INFO:teuthology.orchestra.run.smithi151.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-18T16:06:40.091 INFO:teuthology.orchestra.run.smithi151.stdout: mon_allow_pool_size_one = true 2023-10-18T16:06:40.091 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-10-18T16:06:40.092 INFO:teuthology.orchestra.run.smithi151.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2023-10-18T16:06:40.092 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-10-18T16:06:40.092 INFO:teuthology.orchestra.run.smithi151.stdout: osd default data pool replay window = 5 2023-10-18T16:06:40.092 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-10-18T16:06:40.092 INFO:teuthology.orchestra.run.smithi151.stdout: mon allow pool delete = true 2023-10-18T16:06:40.093 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-10-18T16:06:40.093 INFO:teuthology.orchestra.run.smithi151.stdout: mon cluster log file level = debug 2023-10-18T16:06:40.093 INFO:teuthology.orchestra.run.smithi151.stdout: debug asserts on shutdown = true 2023-10-18T16:06:40.093 INFO:teuthology.orchestra.run.smithi151.stdout: mon health detail to clog = false 2023-10-18T16:06:40.093 INFO:teuthology.orchestra.run.smithi151.stdout: mon host = "172.21.15.88,172.21.15.151,[v2:172.21.15.151:3301,v1:172.21.15.151:6790]" 2023-10-18T16:06:40.093 INFO:teuthology.orchestra.run.smithi151.stdout: ms die on skipped message = False 2023-10-18T16:06:40.094 INFO:teuthology.orchestra.run.smithi151.stdout: fsid = fb1f77e8-ead0-4f90-8128-fdf5137eb16f 2023-10-18T16:06:40.094 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-10-18T16:06:40.094 INFO:teuthology.orchestra.run.smithi151.stdout:[osd] 2023-10-18T16:06:40.094 INFO:teuthology.orchestra.run.smithi151.stdout: osd journal size = 100 2023-10-18T16:06:40.094 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-10-18T16:06:40.095 INFO:teuthology.orchestra.run.smithi151.stdout: osd scrub load threshold = 5.0 2023-10-18T16:06:40.095 INFO:teuthology.orchestra.run.smithi151.stdout: osd scrub max interval = 600 2023-10-18T16:06:40.095 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-10-18T16:06:40.095 INFO:teuthology.orchestra.run.smithi151.stdout: osd recover clone overlap = true 2023-10-18T16:06:40.095 INFO:teuthology.orchestra.run.smithi151.stdout: osd recovery max chunk = 1048576 2023-10-18T16:06:40.095 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-10-18T16:06:40.096 INFO:teuthology.orchestra.run.smithi151.stdout: osd debug shutdown = true 2023-10-18T16:06:40.096 INFO:teuthology.orchestra.run.smithi151.stdout: osd debug op order = true 2023-10-18T16:06:40.096 INFO:teuthology.orchestra.run.smithi151.stdout: osd debug verify stray on activate = true 2023-10-18T16:06:40.096 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-10-18T16:06:40.096 INFO:teuthology.orchestra.run.smithi151.stdout: osd open classes on start = true 2023-10-18T16:06:40.097 INFO:teuthology.orchestra.run.smithi151.stdout: osd debug pg log writeout = true 2023-10-18T16:06:40.097 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-10-18T16:06:40.097 INFO:teuthology.orchestra.run.smithi151.stdout: osd deep scrub update digest min age = 30 2023-10-18T16:06:40.097 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-10-18T16:06:40.097 INFO:teuthology.orchestra.run.smithi151.stdout: osd map max advance = 10 2023-10-18T16:06:40.097 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-10-18T16:06:40.098 INFO:teuthology.orchestra.run.smithi151.stdout: journal zero on create = true 2023-10-18T16:06:40.098 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-10-18T16:06:40.098 INFO:teuthology.orchestra.run.smithi151.stdout: filestore ondisk finisher threads = 3 2023-10-18T16:06:40.098 INFO:teuthology.orchestra.run.smithi151.stdout: filestore apply finisher threads = 3 2023-10-18T16:06:40.098 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-10-18T16:06:40.098 INFO:teuthology.orchestra.run.smithi151.stdout: bdev debug aio = true 2023-10-18T16:06:40.099 INFO:teuthology.orchestra.run.smithi151.stdout: osd debug misdirected ops = true 2023-10-18T16:06:40.099 INFO:teuthology.orchestra.run.smithi151.stdout: bluestore block size = 96636764160 2023-10-18T16:06:40.099 INFO:teuthology.orchestra.run.smithi151.stdout: bluestore compression mode = aggressive 2023-10-18T16:06:40.099 INFO:teuthology.orchestra.run.smithi151.stdout: bluestore fsck on mount = True 2023-10-18T16:06:40.099 INFO:teuthology.orchestra.run.smithi151.stdout: debug bluefs = 20 2023-10-18T16:06:40.100 INFO:teuthology.orchestra.run.smithi151.stdout: debug bluestore = 20 2023-10-18T16:06:40.100 INFO:teuthology.orchestra.run.smithi151.stdout: debug ms = 1 2023-10-18T16:06:40.100 INFO:teuthology.orchestra.run.smithi151.stdout: debug osd = 20 2023-10-18T16:06:40.100 INFO:teuthology.orchestra.run.smithi151.stdout: debug rocksdb = 10 2023-10-18T16:06:40.100 INFO:teuthology.orchestra.run.smithi151.stdout: mon osd backfillfull_ratio = 0.85 2023-10-18T16:06:40.101 INFO:teuthology.orchestra.run.smithi151.stdout: mon osd full ratio = 0.9 2023-10-18T16:06:40.101 INFO:teuthology.orchestra.run.smithi151.stdout: mon osd nearfull ratio = 0.8 2023-10-18T16:06:40.101 INFO:teuthology.orchestra.run.smithi151.stdout: osd failsafe full ratio = 0.95 2023-10-18T16:06:40.101 INFO:teuthology.orchestra.run.smithi151.stdout: osd max markdown count = 1000 2023-10-18T16:06:40.101 INFO:teuthology.orchestra.run.smithi151.stdout: osd objectstore = bluestore 2023-10-18T16:06:40.101 INFO:teuthology.orchestra.run.smithi151.stdout: osd op complaint time = 180 2023-10-18T16:06:40.102 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-10-18T16:06:40.102 INFO:teuthology.orchestra.run.smithi151.stdout:[mgr] 2023-10-18T16:06:40.102 INFO:teuthology.orchestra.run.smithi151.stdout: debug ms = 1 2023-10-18T16:06:40.102 INFO:teuthology.orchestra.run.smithi151.stdout: debug mgr = 20 2023-10-18T16:06:40.102 INFO:teuthology.orchestra.run.smithi151.stdout: debug mon = 20 2023-10-18T16:06:40.103 INFO:teuthology.orchestra.run.smithi151.stdout: debug auth = 20 2023-10-18T16:06:40.103 INFO:teuthology.orchestra.run.smithi151.stdout: mon reweight min pgs per osd = 4 2023-10-18T16:06:40.103 INFO:teuthology.orchestra.run.smithi151.stdout: mon reweight min bytes per osd = 10 2023-10-18T16:06:40.103 INFO:teuthology.orchestra.run.smithi151.stdout: mgr/telemetry/nag = false 2023-10-18T16:06:40.103 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-10-18T16:06:40.103 INFO:teuthology.orchestra.run.smithi151.stdout:[mon] 2023-10-18T16:06:40.104 INFO:teuthology.orchestra.run.smithi151.stdout: debug ms = 1 2023-10-18T16:06:40.104 INFO:teuthology.orchestra.run.smithi151.stdout: debug mon = 20 2023-10-18T16:06:40.104 INFO:teuthology.orchestra.run.smithi151.stdout: debug paxos = 20 2023-10-18T16:06:40.104 INFO:teuthology.orchestra.run.smithi151.stdout: debug auth = 20 2023-10-18T16:06:40.104 INFO:teuthology.orchestra.run.smithi151.stdout: mon data avail warn = 5 2023-10-18T16:06:40.104 INFO:teuthology.orchestra.run.smithi151.stdout: mon mgr mkfs grace = 240 2023-10-18T16:06:40.105 INFO:teuthology.orchestra.run.smithi151.stdout: mon reweight min pgs per osd = 4 2023-10-18T16:06:40.105 INFO:teuthology.orchestra.run.smithi151.stdout: mon osd reporter subtree level = osd 2023-10-18T16:06:40.105 INFO:teuthology.orchestra.run.smithi151.stdout: mon osd prime pg temp = true 2023-10-18T16:06:40.105 INFO:teuthology.orchestra.run.smithi151.stdout: mon reweight min bytes per osd = 10 2023-10-18T16:06:40.105 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-10-18T16:06:40.106 INFO:teuthology.orchestra.run.smithi151.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-18T16:06:40.106 INFO:teuthology.orchestra.run.smithi151.stdout: auth mon ticket ttl = 660 # 11m 2023-10-18T16:06:40.106 INFO:teuthology.orchestra.run.smithi151.stdout: auth service ticket ttl = 240 # 4m 2023-10-18T16:06:40.106 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-10-18T16:06:40.106 INFO:teuthology.orchestra.run.smithi151.stdout: # don't complain about insecure global_id in the test suite 2023-10-18T16:06:40.107 INFO:teuthology.orchestra.run.smithi151.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-18T16:06:40.107 INFO:teuthology.orchestra.run.smithi151.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-18T16:06:40.107 INFO:teuthology.orchestra.run.smithi151.stdout: mon op complaint time = 120 2023-10-18T16:06:40.107 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-10-18T16:06:40.107 INFO:teuthology.orchestra.run.smithi151.stdout:[client] 2023-10-18T16:06:40.108 INFO:teuthology.orchestra.run.smithi151.stdout: rgw cache enabled = true 2023-10-18T16:06:40.108 INFO:teuthology.orchestra.run.smithi151.stdout: rgw enable ops log = true 2023-10-18T16:06:40.108 INFO:teuthology.orchestra.run.smithi151.stdout: rgw enable usage log = true 2023-10-18T16:06:40.108 INFO:teuthology.orchestra.run.smithi151.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-18T16:06:40.108 INFO:teuthology.orchestra.run.smithi151.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-18T16:06:40.109 INFO:teuthology.orchestra.run.smithi151.stdout: client mount timeout = 600 2023-10-18T16:06:40.109 INFO:teuthology.orchestra.run.smithi151.stdout: debug client = 20 2023-10-18T16:06:40.109 INFO:teuthology.orchestra.run.smithi151.stdout: debug ms = 1 2023-10-18T16:06:40.109 INFO:teuthology.orchestra.run.smithi151.stdout: rados mon op timeout = 900 2023-10-18T16:06:40.109 INFO:teuthology.orchestra.run.smithi151.stdout: rados osd op timeout = 900 2023-10-18T16:06:40.109 INFO:teuthology.orchestra.run.smithi151.stdout:[mon.a] 2023-10-18T16:06:40.110 INFO:teuthology.orchestra.run.smithi151.stdout:[mon.b] 2023-10-18T16:06:40.110 INFO:teuthology.orchestra.run.smithi151.stdout:[mon.c] 2023-10-18T16:06:40.110 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.a] 2023-10-18T16:06:40.110 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.c] 2023-10-18T16:06:40.110 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.e] 2023-10-18T16:06:40.111 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.b] 2023-10-18T16:06:40.111 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.d] 2023-10-18T16:06:40.111 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.f] 2023-10-18T16:06:40.111 INFO:teuthology.orchestra.run.smithi151.stdout:[mds] 2023-10-18T16:06:40.111 INFO:teuthology.orchestra.run.smithi151.stdout: debug mds = 20 2023-10-18T16:06:40.112 INFO:teuthology.orchestra.run.smithi151.stdout: debug ms = 1 2023-10-18T16:06:40.112 INFO:teuthology.orchestra.run.smithi151.stdout: mds bal fragment size max = 10000 2023-10-18T16:06:40.112 INFO:teuthology.orchestra.run.smithi151.stdout: mds bal merge size = 5 2023-10-18T16:06:40.112 INFO:teuthology.orchestra.run.smithi151.stdout: mds bal split bits = 3 2023-10-18T16:06:40.112 INFO:teuthology.orchestra.run.smithi151.stdout: mds bal split size = 100 2023-10-18T16:06:40.112 INFO:teuthology.orchestra.run.smithi151.stdout: mds debug frag = True 2023-10-18T16:06:40.113 INFO:teuthology.orchestra.run.smithi151.stdout: mds debug scatterstat = True 2023-10-18T16:06:40.113 INFO:teuthology.orchestra.run.smithi151.stdout: mds oft prefetch dirfrags = False 2023-10-18T16:06:40.113 INFO:teuthology.orchestra.run.smithi151.stdout: mds op complaint time = 180 2023-10-18T16:06:40.113 INFO:teuthology.orchestra.run.smithi151.stdout: mds verify scatter = True 2023-10-18T16:06:40.113 INFO:teuthology.orchestra.run.smithi151.stdout: mds_sleep_rank_change = 5000000.0 2023-10-18T16:06:40.114 INFO:teuthology.orchestra.run.smithi151.stdout: osd op complaint time = 180 2023-10-18T16:06:40.114 INFO:teuthology.orchestra.run.smithi151.stdout: rados mon op timeout = 900 2023-10-18T16:06:40.114 INFO:teuthology.orchestra.run.smithi151.stdout: rados osd op timeout = 900 2023-10-18T16:06:40.114 INFO:tasks.ceph:Creating admin key on mon.a... 2023-10-18T16:06:40.115 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-10-18T16:06:40.174 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-18T16:06:40.175 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-18T16:06:40.175 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-10-18T16:06:40.189 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-18T16:06:40.190 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-10-18T16:06:40.243 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi088.front.sepia.ceph.com 2023-10-18T16:06:40.243 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-18T16:06:40.244 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-18T16:06:40.244 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-18T16:06:40.317 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-18T16:06:40.317 DEBUG:teuthology.orchestra.run.smithi088:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-18T16:06:40.371 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi151.front.sepia.ceph.com 2023-10-18T16:06:40.372 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-10-18T16:06:40.372 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-18T16:06:40.372 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-18T16:06:40.406 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-10-18T16:06:40.406 DEBUG:teuthology.orchestra.run.smithi151:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-18T16:06:40.461 INFO:tasks.ceph:Setting up mon nodes... 2023-10-18T16:06:40.461 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-18T16:06:40.462 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-10-18T16:06:40.508 INFO:teuthology.orchestra.run.smithi088.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-10-18T16:06:40.511 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-10-18T16:06:40.569 INFO:teuthology.orchestra.run.smithi151.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-10-18T16:06:40.571 INFO:tasks.ceph:Setting up mds nodes... 2023-10-18T16:06:40.571 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2023-10-18T16:06:40.618 INFO:teuthology.orchestra.run.smithi088.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2023-10-18T16:06:40.620 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2023-10-18T16:06:40.644 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2023-10-18T16:06:40.730 INFO:teuthology.orchestra.run.smithi088.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2023-10-18T16:06:40.733 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2023-10-18T16:06:40.756 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/mds/ceph-e && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.e /var/lib/ceph/mds/ceph-e/keyring 2023-10-18T16:06:40.846 INFO:teuthology.orchestra.run.smithi088.stdout:creating /var/lib/ceph/mds/ceph-e/keyring 2023-10-18T16:06:40.848 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-e 2023-10-18T16:06:40.914 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2023-10-18T16:06:40.963 INFO:teuthology.orchestra.run.smithi151.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2023-10-18T16:06:40.965 DEBUG:teuthology.orchestra.run.smithi151:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2023-10-18T16:06:40.993 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2023-10-18T16:06:41.079 INFO:teuthology.orchestra.run.smithi151.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2023-10-18T16:06:41.081 DEBUG:teuthology.orchestra.run.smithi151:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2023-10-18T16:06:41.105 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /var/lib/ceph/mds/ceph-f && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.f /var/lib/ceph/mds/ceph-f/keyring 2023-10-18T16:06:41.190 INFO:teuthology.orchestra.run.smithi151.stdout:creating /var/lib/ceph/mds/ceph-f/keyring 2023-10-18T16:06:41.193 DEBUG:teuthology.orchestra.run.smithi151:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-f 2023-10-18T16:06:41.216 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-18T16:06:41.216 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-10-18T16:06:41.253 INFO:teuthology.orchestra.run.smithi088.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-10-18T16:06:41.265 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-18T16:06:41.266 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi088.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi151.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-10-18T16:06:41.266 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-10-18T16:06:41.329 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-18T16:06:41.329 INFO:tasks.ceph:role: osd.0 2023-10-18T16:06:41.330 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi088.front.sepia.ceph.com 2023-10-18T16:06:41.330 DEBUG:teuthology.orchestra.run.smithi088:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-18T16:06:41.400 INFO:teuthology.orchestra.run.smithi088.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-18T16:06:41.400 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-18T16:06:41.401 INFO:teuthology.orchestra.run.smithi088.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-18T16:06:41.401 INFO:teuthology.orchestra.run.smithi088.stdout: = reflink=1 2023-10-18T16:06:41.401 INFO:teuthology.orchestra.run.smithi088.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-18T16:06:41.401 INFO:teuthology.orchestra.run.smithi088.stdout: = sunit=0 swidth=0 blks 2023-10-18T16:06:41.401 INFO:teuthology.orchestra.run.smithi088.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-18T16:06:41.402 INFO:teuthology.orchestra.run.smithi088.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-18T16:06:41.402 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-18T16:06:41.402 INFO:teuthology.orchestra.run.smithi088.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-18T16:06:41.829 INFO:teuthology.orchestra.run.smithi088.stdout:Discarding blocks...Done. 2023-10-18T16:06:41.831 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi088.front.sepia.ceph.com -o noatime 2023-10-18T16:06:41.831 DEBUG:teuthology.orchestra.run.smithi088:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-10-18T16:06:42.008 DEBUG:teuthology.orchestra.run.smithi088:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-10-18T16:06:42.038 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-10-18T16:06:42.103 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-18T16:06:42.103 INFO:tasks.ceph:role: osd.1 2023-10-18T16:06:42.103 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi088.front.sepia.ceph.com 2023-10-18T16:06:42.104 DEBUG:teuthology.orchestra.run.smithi088:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-18T16:06:42.174 INFO:teuthology.orchestra.run.smithi088.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-18T16:06:42.174 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-18T16:06:42.175 INFO:teuthology.orchestra.run.smithi088.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-18T16:06:42.175 INFO:teuthology.orchestra.run.smithi088.stdout: = reflink=1 2023-10-18T16:06:42.175 INFO:teuthology.orchestra.run.smithi088.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-18T16:06:42.175 INFO:teuthology.orchestra.run.smithi088.stdout: = sunit=0 swidth=0 blks 2023-10-18T16:06:42.175 INFO:teuthology.orchestra.run.smithi088.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-18T16:06:42.176 INFO:teuthology.orchestra.run.smithi088.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-18T16:06:42.176 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-18T16:06:42.176 INFO:teuthology.orchestra.run.smithi088.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-18T16:06:42.591 INFO:teuthology.orchestra.run.smithi088.stdout:Discarding blocks...Done. 2023-10-18T16:06:42.592 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi088.front.sepia.ceph.com -o noatime 2023-10-18T16:06:42.593 DEBUG:teuthology.orchestra.run.smithi088:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-10-18T16:06:42.694 DEBUG:teuthology.orchestra.run.smithi088:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-10-18T16:06:42.723 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-10-18T16:06:42.787 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-18T16:06:42.787 INFO:tasks.ceph:role: osd.2 2023-10-18T16:06:42.788 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi088.front.sepia.ceph.com 2023-10-18T16:06:42.788 DEBUG:teuthology.orchestra.run.smithi088:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-18T16:06:42.858 INFO:teuthology.orchestra.run.smithi088.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-18T16:06:42.858 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-18T16:06:42.858 INFO:teuthology.orchestra.run.smithi088.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-18T16:06:42.859 INFO:teuthology.orchestra.run.smithi088.stdout: = reflink=1 2023-10-18T16:06:42.859 INFO:teuthology.orchestra.run.smithi088.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-18T16:06:42.859 INFO:teuthology.orchestra.run.smithi088.stdout: = sunit=0 swidth=0 blks 2023-10-18T16:06:42.859 INFO:teuthology.orchestra.run.smithi088.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-18T16:06:42.860 INFO:teuthology.orchestra.run.smithi088.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-18T16:06:42.860 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-18T16:06:42.860 INFO:teuthology.orchestra.run.smithi088.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-18T16:06:43.297 INFO:teuthology.orchestra.run.smithi088.stdout:Discarding blocks...Done. 2023-10-18T16:06:43.299 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi088.front.sepia.ceph.com -o noatime 2023-10-18T16:06:43.299 DEBUG:teuthology.orchestra.run.smithi088:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-10-18T16:06:43.466 DEBUG:teuthology.orchestra.run.smithi088:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-10-18T16:06:43.497 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-10-18T16:06:43.561 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-18T16:06:43.562 INFO:tasks.ceph:role: osd.3 2023-10-18T16:06:43.562 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi088.front.sepia.ceph.com 2023-10-18T16:06:43.563 DEBUG:teuthology.orchestra.run.smithi088:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-10-18T16:06:43.633 INFO:teuthology.orchestra.run.smithi088.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-10-18T16:06:43.634 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-18T16:06:43.634 INFO:teuthology.orchestra.run.smithi088.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-18T16:06:43.634 INFO:teuthology.orchestra.run.smithi088.stdout: = reflink=1 2023-10-18T16:06:43.634 INFO:teuthology.orchestra.run.smithi088.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-18T16:06:43.635 INFO:teuthology.orchestra.run.smithi088.stdout: = sunit=0 swidth=0 blks 2023-10-18T16:06:43.635 INFO:teuthology.orchestra.run.smithi088.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-18T16:06:43.635 INFO:teuthology.orchestra.run.smithi088.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-18T16:06:43.635 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-18T16:06:43.635 INFO:teuthology.orchestra.run.smithi088.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-18T16:06:44.077 INFO:teuthology.orchestra.run.smithi088.stdout:Discarding blocks...Done. 2023-10-18T16:06:44.079 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi088.front.sepia.ceph.com -o noatime 2023-10-18T16:06:44.079 DEBUG:teuthology.orchestra.run.smithi088:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-10-18T16:06:44.256 DEBUG:teuthology.orchestra.run.smithi088:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-10-18T16:06:44.286 DEBUG:teuthology.orchestra.run.smithi088:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-18T16:06:44.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:06:44.371+0000 7fc89409b200 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-10-18T16:06:44.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:06:44.371+0000 7fc89409b200 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-10-18T16:06:44.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:06:44.371+0000 7fc89409b200 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-18T16:06:44.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:06:44.371+0000 7fc89409b200 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-18T16:06:44.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:06:44.371+0000 7fc89409b200 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-18T16:06:44.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:06:44.372+0000 7fc89409b200 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-10-18T16:06:47.230 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-10-18T16:06:47.256 DEBUG:teuthology.orchestra.run.smithi088:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-18T16:06:47.338 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:06:47.334+0000 7f4a09368200 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-10-18T16:06:47.338 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:06:47.334+0000 7f4a09368200 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-10-18T16:06:47.338 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:06:47.334+0000 7f4a09368200 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-18T16:06:47.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:06:47.334+0000 7f4a09368200 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-18T16:06:47.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:06:47.334+0000 7f4a09368200 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-18T16:06:47.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:06:47.335+0000 7f4a09368200 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-10-18T16:06:50.184 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-10-18T16:06:50.253 DEBUG:teuthology.orchestra.run.smithi088:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-18T16:06:50.338 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:06:50.335+0000 7f999fde9200 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-10-18T16:06:50.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:06:50.335+0000 7f999fde9200 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-10-18T16:06:50.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:06:50.335+0000 7f999fde9200 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-18T16:06:50.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:06:50.335+0000 7f999fde9200 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-18T16:06:50.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:06:50.335+0000 7f999fde9200 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-18T16:06:50.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:06:50.335+0000 7f999fde9200 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-10-18T16:06:53.178 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-10-18T16:06:53.205 DEBUG:teuthology.orchestra.run.smithi088:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-18T16:06:53.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:06:53.283+0000 7f9c2a2ea200 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-10-18T16:06:53.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:06:53.283+0000 7f9c2a2ea200 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-10-18T16:06:53.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:06:53.284+0000 7f9c2a2ea200 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-18T16:06:53.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:06:53.284+0000 7f9c2a2ea200 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-18T16:06:53.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:06:53.284+0000 7f9c2a2ea200 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-18T16:06:53.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:06:53.284+0000 7f9c2a2ea200 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-10-18T16:06:56.128 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-10-18T16:06:56.198 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-10-18T16:06:56.228 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-18T16:06:56.228 INFO:tasks.ceph:role: osd.4 2023-10-18T16:06:56.229 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi151.front.sepia.ceph.com 2023-10-18T16:06:56.229 DEBUG:teuthology.orchestra.run.smithi151:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-18T16:06:56.303 INFO:teuthology.orchestra.run.smithi151.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-18T16:06:56.303 INFO:teuthology.orchestra.run.smithi151.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-18T16:06:56.304 INFO:teuthology.orchestra.run.smithi151.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-18T16:06:56.304 INFO:teuthology.orchestra.run.smithi151.stdout: = reflink=1 2023-10-18T16:06:56.304 INFO:teuthology.orchestra.run.smithi151.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-18T16:06:56.304 INFO:teuthology.orchestra.run.smithi151.stdout: = sunit=0 swidth=0 blks 2023-10-18T16:06:56.304 INFO:teuthology.orchestra.run.smithi151.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-18T16:06:56.305 INFO:teuthology.orchestra.run.smithi151.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-18T16:06:56.305 INFO:teuthology.orchestra.run.smithi151.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-18T16:06:56.305 INFO:teuthology.orchestra.run.smithi151.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-18T16:06:56.676 INFO:teuthology.orchestra.run.smithi151.stdout:Discarding blocks...Done. 2023-10-18T16:06:56.678 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi151.front.sepia.ceph.com -o noatime 2023-10-18T16:06:56.678 DEBUG:teuthology.orchestra.run.smithi151:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-10-18T16:06:56.809 DEBUG:teuthology.orchestra.run.smithi151:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-10-18T16:06:56.841 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-10-18T16:06:56.907 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-18T16:06:56.907 INFO:tasks.ceph:role: osd.5 2023-10-18T16:06:56.907 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi151.front.sepia.ceph.com 2023-10-18T16:06:56.908 DEBUG:teuthology.orchestra.run.smithi151:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-18T16:06:56.981 INFO:teuthology.orchestra.run.smithi151.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-18T16:06:56.981 INFO:teuthology.orchestra.run.smithi151.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-18T16:06:56.981 INFO:teuthology.orchestra.run.smithi151.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-18T16:06:56.981 INFO:teuthology.orchestra.run.smithi151.stdout: = reflink=1 2023-10-18T16:06:56.981 INFO:teuthology.orchestra.run.smithi151.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-18T16:06:56.982 INFO:teuthology.orchestra.run.smithi151.stdout: = sunit=0 swidth=0 blks 2023-10-18T16:06:56.982 INFO:teuthology.orchestra.run.smithi151.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-18T16:06:56.982 INFO:teuthology.orchestra.run.smithi151.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-18T16:06:56.982 INFO:teuthology.orchestra.run.smithi151.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-18T16:06:56.982 INFO:teuthology.orchestra.run.smithi151.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-18T16:06:57.343 INFO:teuthology.orchestra.run.smithi151.stdout:Discarding blocks...Done. 2023-10-18T16:06:57.345 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi151.front.sepia.ceph.com -o noatime 2023-10-18T16:06:57.346 DEBUG:teuthology.orchestra.run.smithi151:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-10-18T16:06:57.431 DEBUG:teuthology.orchestra.run.smithi151:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-10-18T16:06:57.460 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-10-18T16:06:57.524 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-18T16:06:57.524 INFO:tasks.ceph:role: osd.6 2023-10-18T16:06:57.525 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi151.front.sepia.ceph.com 2023-10-18T16:06:57.525 DEBUG:teuthology.orchestra.run.smithi151:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-18T16:06:57.597 INFO:teuthology.orchestra.run.smithi151.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-18T16:06:57.598 INFO:teuthology.orchestra.run.smithi151.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-18T16:06:57.598 INFO:teuthology.orchestra.run.smithi151.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-18T16:06:57.598 INFO:teuthology.orchestra.run.smithi151.stdout: = reflink=1 2023-10-18T16:06:57.598 INFO:teuthology.orchestra.run.smithi151.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-18T16:06:57.598 INFO:teuthology.orchestra.run.smithi151.stdout: = sunit=0 swidth=0 blks 2023-10-18T16:06:57.599 INFO:teuthology.orchestra.run.smithi151.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-18T16:06:57.599 INFO:teuthology.orchestra.run.smithi151.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-18T16:06:57.599 INFO:teuthology.orchestra.run.smithi151.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-18T16:06:57.599 INFO:teuthology.orchestra.run.smithi151.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-18T16:06:57.964 INFO:teuthology.orchestra.run.smithi151.stdout:Discarding blocks...Done. 2023-10-18T16:06:57.966 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi151.front.sepia.ceph.com -o noatime 2023-10-18T16:06:57.966 DEBUG:teuthology.orchestra.run.smithi151:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-10-18T16:06:58.079 DEBUG:teuthology.orchestra.run.smithi151:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-10-18T16:06:58.109 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-10-18T16:06:58.176 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-18T16:06:58.177 INFO:tasks.ceph:role: osd.7 2023-10-18T16:06:58.177 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi151.front.sepia.ceph.com 2023-10-18T16:06:58.177 DEBUG:teuthology.orchestra.run.smithi151:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-10-18T16:06:58.251 INFO:teuthology.orchestra.run.smithi151.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-10-18T16:06:58.251 INFO:teuthology.orchestra.run.smithi151.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-18T16:06:58.251 INFO:teuthology.orchestra.run.smithi151.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-18T16:06:58.251 INFO:teuthology.orchestra.run.smithi151.stdout: = reflink=1 2023-10-18T16:06:58.252 INFO:teuthology.orchestra.run.smithi151.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-18T16:06:58.252 INFO:teuthology.orchestra.run.smithi151.stdout: = sunit=0 swidth=0 blks 2023-10-18T16:06:58.252 INFO:teuthology.orchestra.run.smithi151.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-18T16:06:58.252 INFO:teuthology.orchestra.run.smithi151.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-18T16:06:58.253 INFO:teuthology.orchestra.run.smithi151.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-18T16:06:58.253 INFO:teuthology.orchestra.run.smithi151.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-18T16:06:58.624 INFO:teuthology.orchestra.run.smithi151.stdout:Discarding blocks...Done. 2023-10-18T16:06:58.626 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi151.front.sepia.ceph.com -o noatime 2023-10-18T16:06:58.626 DEBUG:teuthology.orchestra.run.smithi151:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-10-18T16:06:58.746 DEBUG:teuthology.orchestra.run.smithi151:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-10-18T16:06:58.776 DEBUG:teuthology.orchestra.run.smithi151:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-18T16:06:58.859 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:06:58.856+0000 7f8569744200 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-10-18T16:06:58.863 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:06:58.860+0000 7f8569744200 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-10-18T16:06:58.863 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:06:58.860+0000 7f8569744200 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-18T16:06:58.863 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:06:58.860+0000 7f8569744200 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-18T16:06:58.863 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:06:58.860+0000 7f8569744200 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-18T16:06:58.864 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:06:58.861+0000 7f8569744200 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-10-18T16:07:01.727 DEBUG:teuthology.orchestra.run.smithi151:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-10-18T16:07:01.797 DEBUG:teuthology.orchestra.run.smithi151:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-18T16:07:01.883 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:01.880+0000 7f53ef74b200 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-10-18T16:07:01.883 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:01.880+0000 7f53ef74b200 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-10-18T16:07:01.884 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:01.880+0000 7f53ef74b200 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-18T16:07:01.884 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:01.880+0000 7f53ef74b200 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-18T16:07:01.884 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:01.880+0000 7f53ef74b200 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-18T16:07:01.884 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:01.881+0000 7f53ef74b200 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-10-18T16:07:04.746 DEBUG:teuthology.orchestra.run.smithi151:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-10-18T16:07:04.773 DEBUG:teuthology.orchestra.run.smithi151:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-18T16:07:04.857 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:04.854+0000 7f4da7ed0200 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-10-18T16:07:04.857 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:04.854+0000 7f4da7ed0200 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-10-18T16:07:04.858 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:04.854+0000 7f4da7ed0200 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-18T16:07:04.858 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:04.854+0000 7f4da7ed0200 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-18T16:07:04.858 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:04.854+0000 7f4da7ed0200 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-18T16:07:04.858 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:04.855+0000 7f4da7ed0200 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-10-18T16:07:07.734 DEBUG:teuthology.orchestra.run.smithi151:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-10-18T16:07:07.803 DEBUG:teuthology.orchestra.run.smithi151:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-18T16:07:07.886 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:07.883+0000 7f3bac3a1200 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-10-18T16:07:07.887 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:07.884+0000 7f3bac3a1200 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-10-18T16:07:07.887 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:07.884+0000 7f3bac3a1200 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-18T16:07:07.887 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:07.884+0000 7f3bac3a1200 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-18T16:07:07.887 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:07.884+0000 7f3bac3a1200 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-18T16:07:07.888 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:07.884+0000 7f3bac3a1200 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-10-18T16:07:08.036 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-18T16:07:08.038 DEBUG:teuthology.orchestra.run.smithi151:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-18T16:07:10.748 DEBUG:teuthology.orchestra.run.smithi151:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-10-18T16:07:10.776 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-18T16:07:10.777 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-18T16:07:10.777 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-10-18T16:07:10.804 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-18T16:07:10.805 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2023-10-18T16:07:10.870 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-18T16:07:10.871 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2023-10-18T16:07:10.937 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-18T16:07:10.937 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/mds/ceph-e/keyring of=/dev/stdout 2023-10-18T16:07:11.002 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-18T16:07:11.003 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-10-18T16:07:11.067 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-18T16:07:11.067 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-10-18T16:07:11.132 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-18T16:07:11.132 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-10-18T16:07:11.198 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-18T16:07:11.198 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-10-18T16:07:11.262 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-10-18T16:07:11.262 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-10-18T16:07:11.291 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-10-18T16:07:11.291 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2023-10-18T16:07:11.358 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-10-18T16:07:11.359 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2023-10-18T16:07:11.427 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-10-18T16:07:11.427 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/mds/ceph-f/keyring of=/dev/stdout 2023-10-18T16:07:11.493 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-10-18T16:07:11.556 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-10-18T16:07:11.584 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-10-18T16:07:11.585 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-10-18T16:07:11.654 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-10-18T16:07:11.655 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-10-18T16:07:11.721 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-10-18T16:07:11.722 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-10-18T16:07:11.786 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-18T16:07:11.786 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-10-18T16:07:11.801 INFO:tasks.ceph:Adding keys to all mons... 2023-10-18T16:07:11.802 DEBUG:teuthology.orchestra.run.smithi088:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-18T16:07:11.844 DEBUG:teuthology.orchestra.run.smithi151:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-18T16:07:11.864 INFO:teuthology.orchestra.run.smithi088.stdout:[mgr.y] 2023-10-18T16:07:11.864 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQCQAjBlO9QnHhAAdi7tbPsQD08oDgzCMO4kmw== 2023-10-18T16:07:11.864 INFO:teuthology.orchestra.run.smithi088.stdout:[mds.a] 2023-10-18T16:07:11.865 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQCQAjBlsCi4JBAAEJUYnEMpCxfalT5dp5TUdg== 2023-10-18T16:07:11.865 INFO:teuthology.orchestra.run.smithi088.stdout:[mds.c] 2023-10-18T16:07:11.865 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQCQAjBlDKhtKxAArP4pnqsVYuxCWMKdTFeCKA== 2023-10-18T16:07:11.865 INFO:teuthology.orchestra.run.smithi088.stdout:[mds.e] 2023-10-18T16:07:11.865 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQCQAjBlOiBJMhAAre1At+YsY0c+k35KGG0O4w== 2023-10-18T16:07:11.866 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.0] 2023-10-18T16:07:11.866 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQCUAjBl4Vs3FhAAgvt6ycxkK87AxL8nTfBKWQ== 2023-10-18T16:07:11.866 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.1] 2023-10-18T16:07:11.866 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQCXAjBlv2j7ExAAyvZhi4OHlR3dFtK37hV7hg== 2023-10-18T16:07:11.866 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.2] 2023-10-18T16:07:11.867 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQCaAjBlmq8JFBAAXmmh9m326R6e0BySTAxO4w== 2023-10-18T16:07:11.867 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.3] 2023-10-18T16:07:11.867 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQCdAjBlmmb5EBAAznH2oviwUAkUJbtfTc5mag== 2023-10-18T16:07:11.867 INFO:teuthology.orchestra.run.smithi088.stdout:[mgr.x] 2023-10-18T16:07:11.867 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQCQAjBldvTIIRAA3s1JKNBW3V1w1voqQIcDhw== 2023-10-18T16:07:11.868 INFO:teuthology.orchestra.run.smithi088.stdout:[mds.b] 2023-10-18T16:07:11.868 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQCQAjBlfU9BORAAMnlV/DuCEvIIVHb0SHJOKg== 2023-10-18T16:07:11.868 INFO:teuthology.orchestra.run.smithi088.stdout:[mds.d] 2023-10-18T16:07:11.868 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQCRAjBlZkiXBBAAbZf5w+aZkPM4Cy9HgMdc6A== 2023-10-18T16:07:11.869 INFO:teuthology.orchestra.run.smithi088.stdout:[mds.f] 2023-10-18T16:07:11.869 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQCRAjBlTC48CxAA6PPWqRmNizgRrxlYVpYJ7w== 2023-10-18T16:07:11.869 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.4] 2023-10-18T16:07:11.869 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQCiAjBlHJwaMxAAE5zy7vTRlPLLMn471snNUA== 2023-10-18T16:07:11.869 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.5] 2023-10-18T16:07:11.870 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQClAjBlPnd8NBAArDJpxl+XIjhV9S6hArY7Vw== 2023-10-18T16:07:11.870 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.6] 2023-10-18T16:07:11.870 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQCoAjBl+zryMhAALc8Rn6ddV0gFIa9AfwKLjw== 2023-10-18T16:07:11.870 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.7] 2023-10-18T16:07:11.870 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQCrAjBlujmwNBAAmuzVjHClbb4ykr/xAaFpzg== 2023-10-18T16:07:11.871 INFO:teuthology.orchestra.run.smithi088.stdout:[client.0] 2023-10-18T16:07:11.871 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQCRAjBl/R37DhAA+WDvh+M/SSGIgvvfQ3SnZw== 2023-10-18T16:07:11.872 INFO:teuthology.orchestra.run.smithi151.stdout:[mgr.y] 2023-10-18T16:07:11.872 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQCQAjBlO9QnHhAAdi7tbPsQD08oDgzCMO4kmw== 2023-10-18T16:07:11.873 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.a] 2023-10-18T16:07:11.873 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQCQAjBlsCi4JBAAEJUYnEMpCxfalT5dp5TUdg== 2023-10-18T16:07:11.873 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.c] 2023-10-18T16:07:11.873 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQCQAjBlDKhtKxAArP4pnqsVYuxCWMKdTFeCKA== 2023-10-18T16:07:11.873 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.e] 2023-10-18T16:07:11.874 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQCQAjBlOiBJMhAAre1At+YsY0c+k35KGG0O4w== 2023-10-18T16:07:11.874 INFO:teuthology.orchestra.run.smithi151.stdout:[osd.0] 2023-10-18T16:07:11.874 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQCUAjBl4Vs3FhAAgvt6ycxkK87AxL8nTfBKWQ== 2023-10-18T16:07:11.874 INFO:teuthology.orchestra.run.smithi151.stdout:[osd.1] 2023-10-18T16:07:11.874 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQCXAjBlv2j7ExAAyvZhi4OHlR3dFtK37hV7hg== 2023-10-18T16:07:11.875 INFO:teuthology.orchestra.run.smithi151.stdout:[osd.2] 2023-10-18T16:07:11.875 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQCaAjBlmq8JFBAAXmmh9m326R6e0BySTAxO4w== 2023-10-18T16:07:11.875 INFO:teuthology.orchestra.run.smithi151.stdout:[osd.3] 2023-10-18T16:07:11.875 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQCdAjBlmmb5EBAAznH2oviwUAkUJbtfTc5mag== 2023-10-18T16:07:11.875 INFO:teuthology.orchestra.run.smithi151.stdout:[mgr.x] 2023-10-18T16:07:11.876 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQCQAjBldvTIIRAA3s1JKNBW3V1w1voqQIcDhw== 2023-10-18T16:07:11.876 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.b] 2023-10-18T16:07:11.876 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQCQAjBlfU9BORAAMnlV/DuCEvIIVHb0SHJOKg== 2023-10-18T16:07:11.876 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.d] 2023-10-18T16:07:11.876 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQCRAjBlZkiXBBAAbZf5w+aZkPM4Cy9HgMdc6A== 2023-10-18T16:07:11.877 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.f] 2023-10-18T16:07:11.877 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQCRAjBlTC48CxAA6PPWqRmNizgRrxlYVpYJ7w== 2023-10-18T16:07:11.877 INFO:teuthology.orchestra.run.smithi151.stdout:[osd.4] 2023-10-18T16:07:11.877 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQCiAjBlHJwaMxAAE5zy7vTRlPLLMn471snNUA== 2023-10-18T16:07:11.877 INFO:teuthology.orchestra.run.smithi151.stdout:[osd.5] 2023-10-18T16:07:11.878 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQClAjBlPnd8NBAArDJpxl+XIjhV9S6hArY7Vw== 2023-10-18T16:07:11.878 INFO:teuthology.orchestra.run.smithi151.stdout:[osd.6] 2023-10-18T16:07:11.878 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQCoAjBl+zryMhAALc8Rn6ddV0gFIa9AfwKLjw== 2023-10-18T16:07:11.878 INFO:teuthology.orchestra.run.smithi151.stdout:[osd.7] 2023-10-18T16:07:11.878 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQCrAjBlujmwNBAAmuzVjHClbb4ykr/xAaFpzg== 2023-10-18T16:07:11.879 INFO:teuthology.orchestra.run.smithi151.stdout:[client.0] 2023-10-18T16:07:11.879 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQCRAjBl/R37DhAA+WDvh+M/SSGIgvvfQ3SnZw== 2023-10-18T16:07:11.879 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-18T16:07:11.915 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-18T16:07:11.960 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-18T16:07:11.963 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-18T16:07:12.011 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-18T16:07:12.013 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-18T16:07:12.060 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-18T16:07:12.062 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-18T16:07:12.110 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-18T16:07:12.112 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-18T16:07:12.157 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-18T16:07:12.159 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-18T16:07:12.204 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-18T16:07:12.244 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-18T16:07:12.290 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-18T16:07:12.293 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-18T16:07:12.337 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-18T16:07:12.339 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-18T16:07:12.384 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-18T16:07:12.387 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-18T16:07:12.434 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-18T16:07:12.436 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-18T16:07:12.484 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-18T16:07:12.486 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-18T16:07:12.534 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-18T16:07:12.536 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-18T16:07:12.581 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-18T16:07:12.583 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-18T16:07:12.628 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-18T16:07:12.631 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-18T16:07:12.676 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-18T16:07:12.679 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-18T16:07:12.723 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-18T16:07:12.726 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-18T16:07:12.811 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-18T16:07:12.812 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-10-18T16:07:12.836 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-18T16:07:12.927 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-10-18T16:07:12.951 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-10-18T16:07:12.976 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-18T16:07:13.067 DEBUG:teuthology.orchestra.run.smithi151:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-10-18T16:07:13.091 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-10-18T16:07:13.153 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-18T16:07:13.242 DEBUG:teuthology.orchestra.run.smithi151:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-10-18T16:07:13.266 DEBUG:teuthology.orchestra.run.smithi088:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-18T16:07:13.268 DEBUG:teuthology.orchestra.run.smithi151:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-18T16:07:13.320 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-10-18T16:07:13.321 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-18T16:07:13.321 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-10-18T16:07:13.323 INFO:tasks.ceph.mon.a:Started 2023-10-18T16:07:13.323 INFO:tasks.ceph.mon.b:Restarting daemon 2023-10-18T16:07:13.324 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-10-18T16:07:13.362 INFO:tasks.ceph.mon.b:Started 2023-10-18T16:07:13.363 INFO:tasks.ceph.mon.c:Restarting daemon 2023-10-18T16:07:13.363 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-10-18T16:07:13.365 INFO:tasks.ceph.mon.c:Started 2023-10-18T16:07:13.365 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-10-18T16:07:13.366 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-10-18T16:07:13.366 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-10-18T16:07:13.368 INFO:tasks.ceph.mgr.y:Started 2023-10-18T16:07:13.369 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-10-18T16:07:13.369 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-10-18T16:07:13.371 INFO:tasks.ceph.mgr.x:Started 2023-10-18T16:07:13.372 INFO:tasks.ceph:Setting crush tunables to default 2023-10-18T16:07:13.372 DEBUG:teuthology.orchestra.run.smithi088:> sudo ceph --cluster ceph osd crush tunables default 2023-10-18T16:07:13.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:13.484+0000 7f40f2ab9700 1 Processor -- start 2023-10-18T16:07:13.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:13.484+0000 7f40f2ab9700 1 -- start start 2023-10-18T16:07:13.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:13.485+0000 7f40f2ab9700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f40ec149e80 0x7f40ec14a260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:13.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:13.485+0000 7f40f2ab9700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40ec0bb980 con 0x7f40ec0c66d0 2023-10-18T16:07:13.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:13.485+0000 7f40f2ab9700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40ec0bbc60 con 0x7f40ec150b70 2023-10-18T16:07:13.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:13.485+0000 7f40f2ab9700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40ec064b60 con 0x7f40ec149e80 2023-10-18T16:07:13.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:13.485+0000 7f40f22b8700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f40ec0c66d0 0x7f40ec0c2340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:46062/0 (socket says 172.21.15.88:46062) 2023-10-18T16:07:13.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:13.485+0000 7f40f22b8700 1 -- 172.21.15.88:0/2213837317 learned_addr learned my addr 172.21.15.88:0/2213837317 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:07:13.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:13.485+0000 7f40f1ab7700 1 --2- 172.21.15.88:0/2213837317 >> v2:172.21.15.151:3300/0 conn(0x7f40ec149e80 0x7f40ec14a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:13.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:13.485+0000 7f40f1ab7700 1 -- 172.21.15.88:0/2213837317 >> v2:172.21.15.151:3300/0 conn(0x7f40ec149e80 msgr2=0x7f40ec14a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-18T16:07:13.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:13.485+0000 7f40f1ab7700 1 -- 172.21.15.88:0/2213837317 >> v2:172.21.15.151:3300/0 conn(0x7f40ec149e80 msgr2=0x7f40ec14a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-18T16:07:13.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:13.485+0000 7f40f1ab7700 1 --2- 172.21.15.88:0/2213837317 >> v2:172.21.15.151:3300/0 conn(0x7f40ec149e80 0x7f40ec14a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-10-18T16:07:13.492 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:13.485+0000 7f40f1ab7700 1 --2- 172.21.15.88:0/2213837317 >> v2:172.21.15.151:3300/0 conn(0x7f40ec149e80 0x7f40ec14a260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:07:13.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:13.686+0000 7f40f1ab7700 1 --2- 172.21.15.88:0/2213837317 >> v2:172.21.15.151:3300/0 conn(0x7f40ec149e80 0x7f40ec14a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:13.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:13.686+0000 7f40f1ab7700 1 -- 172.21.15.88:0/2213837317 >> v2:172.21.15.151:3300/0 conn(0x7f40ec149e80 msgr2=0x7f40ec14a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-18T16:07:13.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:13.686+0000 7f40f1ab7700 1 -- 172.21.15.88:0/2213837317 >> v2:172.21.15.151:3300/0 conn(0x7f40ec149e80 msgr2=0x7f40ec14a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-18T16:07:13.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:13.686+0000 7f40f1ab7700 1 --2- 172.21.15.88:0/2213837317 >> v2:172.21.15.151:3300/0 conn(0x7f40ec149e80 0x7f40ec14a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-10-18T16:07:13.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:13.686+0000 7f40f1ab7700 1 --2- 172.21.15.88:0/2213837317 >> v2:172.21.15.151:3300/0 conn(0x7f40ec149e80 0x7f40ec14a260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:07:14.091 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:14.087+0000 7f40f1ab7700 1 --2- 172.21.15.88:0/2213837317 >> v2:172.21.15.151:3300/0 conn(0x7f40ec149e80 0x7f40ec14a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:14.091 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:14.087+0000 7f40f1ab7700 1 -- 172.21.15.88:0/2213837317 >> v2:172.21.15.151:3300/0 conn(0x7f40ec149e80 msgr2=0x7f40ec14a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-18T16:07:14.091 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:14.087+0000 7f40f1ab7700 1 -- 172.21.15.88:0/2213837317 >> v2:172.21.15.151:3300/0 conn(0x7f40ec149e80 msgr2=0x7f40ec14a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-18T16:07:14.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:14.087+0000 7f40f1ab7700 1 --2- 172.21.15.88:0/2213837317 >> v2:172.21.15.151:3300/0 conn(0x7f40ec149e80 0x7f40ec14a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-10-18T16:07:14.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:14.087+0000 7f40f1ab7700 1 --2- 172.21.15.88:0/2213837317 >> v2:172.21.15.151:3300/0 conn(0x7f40ec149e80 0x7f40ec14a260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:07:14.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:14.888+0000 7f40f1ab7700 1 --2- 172.21.15.88:0/2213837317 >> v2:172.21.15.151:3300/0 conn(0x7f40ec149e80 0x7f40ec14a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:14.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:14.888+0000 7f40f1ab7700 1 -- 172.21.15.88:0/2213837317 >> v2:172.21.15.151:3300/0 conn(0x7f40ec149e80 msgr2=0x7f40ec14a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-18T16:07:14.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:14.888+0000 7f40f1ab7700 1 -- 172.21.15.88:0/2213837317 >> v2:172.21.15.151:3300/0 conn(0x7f40ec149e80 msgr2=0x7f40ec14a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-18T16:07:14.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:14.888+0000 7f40f1ab7700 1 --2- 172.21.15.88:0/2213837317 >> v2:172.21.15.151:3300/0 conn(0x7f40ec149e80 0x7f40ec14a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-10-18T16:07:14.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:14.888+0000 7f40f1ab7700 1 --2- 172.21.15.88:0/2213837317 >> v2:172.21.15.151:3300/0 conn(0x7f40ec149e80 0x7f40ec14a260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:07:16.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:16.485+0000 7f40e37fe700 1 -- 172.21.15.88:0/2213837317 >> v2:172.21.15.151:3300/0 conn(0x7f40ec149e80 msgr2=0x7f40ec14a260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:07:16.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:16.485+0000 7f40e37fe700 1 --2- 172.21.15.88:0/2213837317 >> v2:172.21.15.151:3300/0 conn(0x7f40ec149e80 0x7f40ec14a260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:16.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:16.485+0000 7f40e37fe700 1 -- 172.21.15.88:0/2213837317 >> v1:172.21.15.151:6789/0 conn(0x7f40ec150b70 legacy=0x7f40ec0c1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:16.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:16.485+0000 7f40e37fe700 1 -- 172.21.15.88:0/2213837317 >> v1:172.21.15.88:6789/0 conn(0x7f40ec0c66d0 legacy=0x7f40ec0c2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:16.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:16.485+0000 7f40e37fe700 1 --2- 172.21.15.88:0/2213837317 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40d00035e0 0x7f40d0005a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:16.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:16.485+0000 7f40e37fe700 1 --2- 172.21.15.88:0/2213837317 >> v2:172.21.15.88:3300/0 conn(0x7f40d0006070 0x7f40d0008530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:16.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:16.485+0000 7f40e37fe700 1 -- 172.21.15.88:0/2213837317 --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40ec064b60 con 0x7f40d0006070 2023-10-18T16:07:16.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:16.485+0000 7f40e37fe700 1 -- 172.21.15.88:0/2213837317 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40d0008b90 con 0x7f40d00035e0 2023-10-18T16:07:16.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:16.485+0000 7f40f12b6700 1 --2- 172.21.15.88:0/2213837317 >> v2:172.21.15.88:3300/0 conn(0x7f40d0006070 0x7f40d0008530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:16.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:16.485+0000 7f40f12b6700 1 -- 172.21.15.88:0/2213837317 >> v2:172.21.15.88:3300/0 conn(0x7f40d0006070 msgr2=0x7f40d0008530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-18T16:07:16.492 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:16.485+0000 7f40f12b6700 1 -- 172.21.15.88:0/2213837317 >> v2:172.21.15.88:3300/0 conn(0x7f40d0006070 msgr2=0x7f40d0008530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-18T16:07:16.492 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:16.485+0000 7f40f12b6700 1 --2- 172.21.15.88:0/2213837317 >> v2:172.21.15.88:3300/0 conn(0x7f40d0006070 0x7f40d0008530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-10-18T16:07:16.492 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:16.485+0000 7f40f12b6700 1 --2- 172.21.15.88:0/2213837317 >> v2:172.21.15.88:3300/0 conn(0x7f40d0006070 0x7f40d0008530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:07:16.492 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:16.485+0000 7f40f1ab7700 1 --2- 172.21.15.88:0/2213837317 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40d00035e0 0x7f40d0005a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:16.492 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:16.486+0000 7f40f1ab7700 1 -- 172.21.15.88:0/2213837317 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40d00035e0 msgr2=0x7f40d0005a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-18T16:07:16.493 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:16.486+0000 7f40f1ab7700 1 -- 172.21.15.88:0/2213837317 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40d00035e0 msgr2=0x7f40d0005a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-18T16:07:16.493 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:16.486+0000 7f40f1ab7700 1 --2- 172.21.15.88:0/2213837317 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40d00035e0 0x7f40d0005a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-10-18T16:07:16.493 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:16.486+0000 7f40f1ab7700 1 --2- 172.21.15.88:0/2213837317 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40d00035e0 0x7f40d0005a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:07:16.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:16.687+0000 7f40f12b6700 1 --2- 172.21.15.88:0/2213837317 >> v2:172.21.15.88:3300/0 conn(0x7f40d0006070 0x7f40d0008530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:16.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:16.687+0000 7f40f12b6700 1 -- 172.21.15.88:0/2213837317 >> v2:172.21.15.88:3300/0 conn(0x7f40d0006070 msgr2=0x7f40d0008530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-18T16:07:16.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:16.687+0000 7f40f12b6700 1 -- 172.21.15.88:0/2213837317 >> v2:172.21.15.88:3300/0 conn(0x7f40d0006070 msgr2=0x7f40d0008530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-18T16:07:16.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:16.687+0000 7f40f12b6700 1 --2- 172.21.15.88:0/2213837317 >> v2:172.21.15.88:3300/0 conn(0x7f40d0006070 0x7f40d0008530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-10-18T16:07:16.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:16.687+0000 7f40f12b6700 1 --2- 172.21.15.88:0/2213837317 >> v2:172.21.15.88:3300/0 conn(0x7f40d0006070 0x7f40d0008530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:07:16.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:16.687+0000 7f40f1ab7700 1 --2- 172.21.15.88:0/2213837317 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40d00035e0 0x7f40d0005a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:16.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:16.687+0000 7f40f1ab7700 1 -- 172.21.15.88:0/2213837317 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40d00035e0 msgr2=0x7f40d0005a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-18T16:07:16.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:16.687+0000 7f40f1ab7700 1 -- 172.21.15.88:0/2213837317 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40d00035e0 msgr2=0x7f40d0005a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-18T16:07:16.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:16.687+0000 7f40f1ab7700 1 --2- 172.21.15.88:0/2213837317 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40d00035e0 0x7f40d0005a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-10-18T16:07:16.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:16.687+0000 7f40f1ab7700 1 --2- 172.21.15.88:0/2213837317 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40d00035e0 0x7f40d0005a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:07:17.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:17.088+0000 7f40f12b6700 1 --2- 172.21.15.88:0/2213837317 >> v2:172.21.15.88:3300/0 conn(0x7f40d0006070 0x7f40d0008530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:17.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:17.088+0000 7f40f12b6700 1 -- 172.21.15.88:0/2213837317 >> v2:172.21.15.88:3300/0 conn(0x7f40d0006070 msgr2=0x7f40d0008530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-18T16:07:17.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:17.088+0000 7f40f12b6700 1 -- 172.21.15.88:0/2213837317 >> v2:172.21.15.88:3300/0 conn(0x7f40d0006070 msgr2=0x7f40d0008530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-18T16:07:17.093 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:17.088+0000 7f40f12b6700 1 --2- 172.21.15.88:0/2213837317 >> v2:172.21.15.88:3300/0 conn(0x7f40d0006070 0x7f40d0008530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-10-18T16:07:17.093 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:17.088+0000 7f40f12b6700 1 --2- 172.21.15.88:0/2213837317 >> v2:172.21.15.88:3300/0 conn(0x7f40d0006070 0x7f40d0008530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:07:17.093 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:17.088+0000 7f40f1ab7700 1 --2- 172.21.15.88:0/2213837317 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40d00035e0 0x7f40d0005a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:17.093 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:17.088+0000 7f40f1ab7700 1 -- 172.21.15.88:0/2213837317 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40d00035e0 msgr2=0x7f40d0005a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-18T16:07:17.093 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:17.088+0000 7f40f1ab7700 1 -- 172.21.15.88:0/2213837317 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40d00035e0 msgr2=0x7f40d0005a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-18T16:07:17.093 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:17.088+0000 7f40f1ab7700 1 --2- 172.21.15.88:0/2213837317 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40d00035e0 0x7f40d0005a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-10-18T16:07:17.094 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:17.088+0000 7f40f1ab7700 1 --2- 172.21.15.88:0/2213837317 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40d00035e0 0x7f40d0005a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:07:17.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:17.889+0000 7f40f12b6700 1 --2- 172.21.15.88:0/2213837317 >> v2:172.21.15.88:3300/0 conn(0x7f40d0006070 0x7f40d0008530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:17.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:17.889+0000 7f40f12b6700 1 -- 172.21.15.88:0/2213837317 >> v2:172.21.15.88:3300/0 conn(0x7f40d0006070 msgr2=0x7f40d0008530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-18T16:07:17.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:17.889+0000 7f40f12b6700 1 -- 172.21.15.88:0/2213837317 >> v2:172.21.15.88:3300/0 conn(0x7f40d0006070 msgr2=0x7f40d0008530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-18T16:07:17.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:17.889+0000 7f40f12b6700 1 --2- 172.21.15.88:0/2213837317 >> v2:172.21.15.88:3300/0 conn(0x7f40d0006070 0x7f40d0008530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-10-18T16:07:17.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:17.889+0000 7f40f1ab7700 1 --2- 172.21.15.88:0/2213837317 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40d00035e0 0x7f40d0005a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:17.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:17.889+0000 7f40f12b6700 1 --2- 172.21.15.88:0/2213837317 >> v2:172.21.15.88:3300/0 conn(0x7f40d0006070 0x7f40d0008530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:07:17.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:17.889+0000 7f40f1ab7700 1 -- 172.21.15.88:0/2213837317 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40d00035e0 msgr2=0x7f40d0005a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-18T16:07:17.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:17.889+0000 7f40f1ab7700 1 -- 172.21.15.88:0/2213837317 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40d00035e0 msgr2=0x7f40d0005a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-18T16:07:17.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:17.889+0000 7f40f1ab7700 1 --2- 172.21.15.88:0/2213837317 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40d00035e0 0x7f40d0005a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-10-18T16:07:17.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:17.889+0000 7f40f1ab7700 1 --2- 172.21.15.88:0/2213837317 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40d00035e0 0x7f40d0005a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:07:19.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:19.485+0000 7f40e37fe700 1 -- 172.21.15.88:0/2213837317 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40d00035e0 msgr2=0x7f40d0005a30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:07:19.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:19.485+0000 7f40e37fe700 1 --2- 172.21.15.88:0/2213837317 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40d00035e0 0x7f40d0005a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:19.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:19.485+0000 7f40e37fe700 1 -- 172.21.15.88:0/2213837317 >> v2:172.21.15.88:3300/0 conn(0x7f40d0006070 msgr2=0x7f40d0008530 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:07:19.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:19.485+0000 7f40e37fe700 1 --2- 172.21.15.88:0/2213837317 >> v2:172.21.15.88:3300/0 conn(0x7f40d0006070 0x7f40d0008530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:19.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:19.485+0000 7f40f22b8700 1 -- 172.21.15.88:0/2213837317 reap_dead start 2023-10-18T16:07:19.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:19.485+0000 7f40e37fe700 1 --2- 172.21.15.88:0/2213837317 >> v2:172.21.15.151:3300/0 conn(0x7f40d0003a00 0x7f40d00064a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:19.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:19.485+0000 7f40e37fe700 1 --2- 172.21.15.88:0/2213837317 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40d0006990 0x7f40d0006e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:19.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:19.485+0000 7f40e37fe700 1 -- 172.21.15.88:0/2213837317 --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40ec0bbf40 con 0x7f40ec150b70 2023-10-18T16:07:19.492 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:19.485+0000 7f40e37fe700 1 -- 172.21.15.88:0/2213837317 --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40ec064b60 con 0x7f40d0003a00 2023-10-18T16:07:19.492 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:19.485+0000 7f40e37fe700 1 -- 172.21.15.88:0/2213837317 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40d0008b90 con 0x7f40d0006990 2023-10-18T16:07:19.492 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:19.486+0000 7f40f22b8700 1 --2- 172.21.15.88:0/2213837317 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40d0006990 0x7f40d0006e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:19.492 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:19.486+0000 7f40f12b6700 1 --2- 172.21.15.88:0/2213837317 >> v2:172.21.15.151:3300/0 conn(0x7f40d0003a00 0x7f40d00064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:19.492 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:19.486+0000 7f40f22b8700 1 -- 172.21.15.88:0/2213837317 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40d0006990 msgr2=0x7f40d0006e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-18T16:07:19.493 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:19.486+0000 7f40f22b8700 1 -- 172.21.15.88:0/2213837317 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40d0006990 msgr2=0x7f40d0006e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-18T16:07:19.493 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:19.486+0000 7f40f22b8700 1 --2- 172.21.15.88:0/2213837317 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40d0006990 0x7f40d0006e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-10-18T16:07:19.493 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:19.486+0000 7f40f12b6700 1 -- 172.21.15.88:0/2213837317 >> v2:172.21.15.151:3300/0 conn(0x7f40d0003a00 msgr2=0x7f40d00064a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-18T16:07:19.493 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:19.486+0000 7f40f12b6700 1 -- 172.21.15.88:0/2213837317 >> v2:172.21.15.151:3300/0 conn(0x7f40d0003a00 msgr2=0x7f40d00064a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-18T16:07:19.493 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:19.486+0000 7f40f12b6700 1 --2- 172.21.15.88:0/2213837317 >> v2:172.21.15.151:3300/0 conn(0x7f40d0003a00 0x7f40d00064a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-10-18T16:07:19.494 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:19.486+0000 7f40f22b8700 1 --2- 172.21.15.88:0/2213837317 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40d0006990 0x7f40d0006e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:07:19.494 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:19.486+0000 7f40f12b6700 1 --2- 172.21.15.88:0/2213837317 >> v2:172.21.15.151:3300/0 conn(0x7f40d0003a00 0x7f40d00064a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:07:19.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:19.687+0000 7f40f12b6700 1 --2- 172.21.15.88:0/2213837317 >> v2:172.21.15.151:3300/0 conn(0x7f40d0003a00 0x7f40d00064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:19.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:19.687+0000 7f40f22b8700 1 --2- 172.21.15.88:0/2213837317 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40d0006990 0x7f40d0006e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:19.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:19.687+0000 7f40f12b6700 1 -- 172.21.15.88:0/2213837317 >> v2:172.21.15.151:3300/0 conn(0x7f40d0003a00 msgr2=0x7f40d00064a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-18T16:07:19.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:19.687+0000 7f40f22b8700 1 -- 172.21.15.88:0/2213837317 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40d0006990 msgr2=0x7f40d0006e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-18T16:07:19.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:19.687+0000 7f40f12b6700 1 -- 172.21.15.88:0/2213837317 >> v2:172.21.15.151:3300/0 conn(0x7f40d0003a00 msgr2=0x7f40d00064a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-18T16:07:19.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:19.687+0000 7f40f22b8700 1 -- 172.21.15.88:0/2213837317 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40d0006990 msgr2=0x7f40d0006e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-18T16:07:19.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:19.687+0000 7f40f22b8700 1 --2- 172.21.15.88:0/2213837317 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40d0006990 0x7f40d0006e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-10-18T16:07:19.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:19.687+0000 7f40f12b6700 1 --2- 172.21.15.88:0/2213837317 >> v2:172.21.15.151:3300/0 conn(0x7f40d0003a00 0x7f40d00064a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-10-18T16:07:19.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:19.687+0000 7f40f12b6700 1 --2- 172.21.15.88:0/2213837317 >> v2:172.21.15.151:3300/0 conn(0x7f40d0003a00 0x7f40d00064a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:07:19.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:19.687+0000 7f40f22b8700 1 --2- 172.21.15.88:0/2213837317 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40d0006990 0x7f40d0006e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:07:20.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:20.088+0000 7f40f12b6700 1 --2- 172.21.15.88:0/2213837317 >> v2:172.21.15.151:3300/0 conn(0x7f40d0003a00 0x7f40d00064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:20.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:20.088+0000 7f40f22b8700 1 --2- 172.21.15.88:0/2213837317 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40d0006990 0x7f40d0006e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:20.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:20.088+0000 7f40f12b6700 1 -- 172.21.15.88:0/2213837317 >> v2:172.21.15.151:3300/0 conn(0x7f40d0003a00 msgr2=0x7f40d00064a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-18T16:07:20.093 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:20.088+0000 7f40f22b8700 1 -- 172.21.15.88:0/2213837317 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40d0006990 msgr2=0x7f40d0006e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-18T16:07:20.093 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:20.088+0000 7f40f12b6700 1 -- 172.21.15.88:0/2213837317 >> v2:172.21.15.151:3300/0 conn(0x7f40d0003a00 msgr2=0x7f40d00064a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-18T16:07:20.093 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:20.088+0000 7f40f22b8700 1 -- 172.21.15.88:0/2213837317 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40d0006990 msgr2=0x7f40d0006e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-18T16:07:20.093 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:20.088+0000 7f40f22b8700 1 --2- 172.21.15.88:0/2213837317 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40d0006990 0x7f40d0006e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-10-18T16:07:20.093 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:20.088+0000 7f40f12b6700 1 --2- 172.21.15.88:0/2213837317 >> v2:172.21.15.151:3300/0 conn(0x7f40d0003a00 0x7f40d00064a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-10-18T16:07:20.094 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:20.088+0000 7f40f12b6700 1 --2- 172.21.15.88:0/2213837317 >> v2:172.21.15.151:3300/0 conn(0x7f40d0003a00 0x7f40d00064a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:07:20.094 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:20.088+0000 7f40f22b8700 1 --2- 172.21.15.88:0/2213837317 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40d0006990 0x7f40d0006e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:07:20.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:20.889+0000 7f40f12b6700 1 --2- 172.21.15.88:0/2213837317 >> v2:172.21.15.151:3300/0 conn(0x7f40d0003a00 0x7f40d00064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:20.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:20.889+0000 7f40f22b8700 1 --2- 172.21.15.88:0/2213837317 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40d0006990 0x7f40d0006e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:20.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:20.889+0000 7f40f12b6700 1 -- 172.21.15.88:0/2213837317 >> v2:172.21.15.151:3300/0 conn(0x7f40d0003a00 msgr2=0x7f40d00064a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-18T16:07:20.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:20.889+0000 7f40f22b8700 1 -- 172.21.15.88:0/2213837317 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40d0006990 msgr2=0x7f40d0006e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-18T16:07:20.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:20.889+0000 7f40f12b6700 1 -- 172.21.15.88:0/2213837317 >> v2:172.21.15.151:3300/0 conn(0x7f40d0003a00 msgr2=0x7f40d00064a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-18T16:07:20.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:20.889+0000 7f40f22b8700 1 -- 172.21.15.88:0/2213837317 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40d0006990 msgr2=0x7f40d0006e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-18T16:07:20.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:20.889+0000 7f40f12b6700 1 --2- 172.21.15.88:0/2213837317 >> v2:172.21.15.151:3300/0 conn(0x7f40d0003a00 0x7f40d00064a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-10-18T16:07:20.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:20.889+0000 7f40f22b8700 1 --2- 172.21.15.88:0/2213837317 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40d0006990 0x7f40d0006e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-10-18T16:07:20.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:20.890+0000 7f40f12b6700 1 --2- 172.21.15.88:0/2213837317 >> v2:172.21.15.151:3300/0 conn(0x7f40d0003a00 0x7f40d00064a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:07:20.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:20.890+0000 7f40f22b8700 1 --2- 172.21.15.88:0/2213837317 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40d0006990 0x7f40d0006e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:07:22.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:22.485+0000 7f40e37fe700 1 -- 172.21.15.88:0/2213837317 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40d0006990 msgr2=0x7f40d0006e10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:07:22.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:22.485+0000 7f40e37fe700 1 --2- 172.21.15.88:0/2213837317 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40d0006990 0x7f40d0006e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:22.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:22.485+0000 7f40e37fe700 1 -- 172.21.15.88:0/2213837317 >> v2:172.21.15.151:3300/0 conn(0x7f40d0003a00 msgr2=0x7f40d00064a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:07:22.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:22.485+0000 7f40e37fe700 1 --2- 172.21.15.88:0/2213837317 >> v2:172.21.15.151:3300/0 conn(0x7f40d0003a00 0x7f40d00064a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:22.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:22.485+0000 7f40e37fe700 1 -- 172.21.15.88:0/2213837317 >> v1:172.21.15.151:6789/0 conn(0x7f40ec150b70 legacy=0x7f40d000b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:22.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:22.485+0000 7f40e37fe700 1 --2- 172.21.15.88:0/2213837317 >> v2:172.21.15.88:3300/0 conn(0x7f40d0008110 0x7f40d0013790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:22.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:22.486+0000 7f40e37fe700 1 -- 172.21.15.88:0/2213837317 --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40d00085e0 con 0x7f40d0007500 2023-10-18T16:07:22.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:22.486+0000 7f40e37fe700 1 -- 172.21.15.88:0/2213837317 --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40ec064b60 con 0x7f40d0008110 2023-10-18T16:07:22.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:22.486+0000 7f40f12b6700 1 --2- 172.21.15.88:0/2213837317 >> v2:172.21.15.88:3300/0 conn(0x7f40d0008110 0x7f40d0013790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:22.492 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:22.486+0000 7f40f12b6700 1 -- 172.21.15.88:0/2213837317 >> v2:172.21.15.88:3300/0 conn(0x7f40d0008110 msgr2=0x7f40d0013790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-18T16:07:22.492 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:22.486+0000 7f40f12b6700 1 -- 172.21.15.88:0/2213837317 >> v2:172.21.15.88:3300/0 conn(0x7f40d0008110 msgr2=0x7f40d0013790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-18T16:07:22.492 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:22.486+0000 7f40f12b6700 1 --2- 172.21.15.88:0/2213837317 >> v2:172.21.15.88:3300/0 conn(0x7f40d0008110 0x7f40d0013790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-10-18T16:07:22.492 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:22.486+0000 7f40f12b6700 1 --2- 172.21.15.88:0/2213837317 >> v2:172.21.15.88:3300/0 conn(0x7f40d0008110 0x7f40d0013790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:07:22.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:22.687+0000 7f40f12b6700 1 --2- 172.21.15.88:0/2213837317 >> v2:172.21.15.88:3300/0 conn(0x7f40d0008110 0x7f40d0013790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:22.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:22.687+0000 7f40f12b6700 1 -- 172.21.15.88:0/2213837317 >> v2:172.21.15.88:3300/0 conn(0x7f40d0008110 msgr2=0x7f40d0013790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-18T16:07:22.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:22.687+0000 7f40f12b6700 1 -- 172.21.15.88:0/2213837317 >> v2:172.21.15.88:3300/0 conn(0x7f40d0008110 msgr2=0x7f40d0013790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-18T16:07:22.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:22.687+0000 7f40f12b6700 1 --2- 172.21.15.88:0/2213837317 >> v2:172.21.15.88:3300/0 conn(0x7f40d0008110 0x7f40d0013790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-10-18T16:07:22.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:22.687+0000 7f40f12b6700 1 --2- 172.21.15.88:0/2213837317 >> v2:172.21.15.88:3300/0 conn(0x7f40d0008110 0x7f40d0013790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:07:23.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.088+0000 7f40f12b6700 1 --2- 172.21.15.88:0/2213837317 >> v2:172.21.15.88:3300/0 conn(0x7f40d0008110 0x7f40d0013790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:23.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.088+0000 7f40f12b6700 1 -- 172.21.15.88:0/2213837317 >> v2:172.21.15.88:3300/0 conn(0x7f40d0008110 msgr2=0x7f40d0013790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-18T16:07:23.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.088+0000 7f40f12b6700 1 -- 172.21.15.88:0/2213837317 >> v2:172.21.15.88:3300/0 conn(0x7f40d0008110 msgr2=0x7f40d0013790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-18T16:07:23.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.088+0000 7f40f12b6700 1 --2- 172.21.15.88:0/2213837317 >> v2:172.21.15.88:3300/0 conn(0x7f40d0008110 0x7f40d0013790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-10-18T16:07:23.093 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.088+0000 7f40f12b6700 1 --2- 172.21.15.88:0/2213837317 >> v2:172.21.15.88:3300/0 conn(0x7f40d0008110 0x7f40d0013790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:07:23.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.682+0000 7f40f0ab5700 1 -- 172.21.15.88:0/2213837317 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 520434481 0 0) 0x7f40d00085e0 con 0x7f40d0007500 2023-10-18T16:07:23.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.682+0000 7f40f0ab5700 1 -- 172.21.15.88:0/2213837317 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40d80033f0 con 0x7f40d0007500 2023-10-18T16:07:23.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.682+0000 7f40f0ab5700 1 -- 172.21.15.88:0/2213837317 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 670367165 0 0) 0x7f40d80033f0 con 0x7f40d0007500 2023-10-18T16:07:23.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.682+0000 7f40f0ab5700 1 -- 172.21.15.88:0/2213837317 >> v2:172.21.15.88:3300/0 conn(0x7f40d0008110 msgr2=0x7f40d0013790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:07:23.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.682+0000 7f40f0ab5700 1 --2- 172.21.15.88:0/2213837317 >> v2:172.21.15.88:3300/0 conn(0x7f40d0008110 0x7f40d0013790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:23.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.682+0000 7f40f0ab5700 1 -- 172.21.15.88:0/2213837317 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40d0014e50 con 0x7f40d0007500 2023-10-18T16:07:23.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.682+0000 7f40f0ab5700 1 -- 172.21.15.88:0/2213837317 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441715075 0 0) 0x7f40e8015500 con 0x7f40d0007500 2023-10-18T16:07:23.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.682+0000 7f40f0ab5700 1 -- 172.21.15.88:0/2213837317 >> v1:172.21.15.88:6789/0 conn(0x7f40d0007500 legacy=0x7f40d0007910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:23.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.682+0000 7f40f22b8700 1 -- 172.21.15.88:0/2213837317 reap_dead start 2023-10-18T16:07:23.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.682+0000 7f40f0ab5700 1 --2- 172.21.15.88:0/2213837317 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40d80038e0 0x7f40d8003ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:23.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.682+0000 7f40f0ab5700 1 --2- 172.21.15.88:0/2213837317 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40d8004690 0x7f40d8004b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:23.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.682+0000 7f40f0ab5700 1 --2- 172.21.15.88:0/2213837317 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40d8005050 0x7f40d800b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:23.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.682+0000 7f40f0ab5700 1 -- 172.21.15.88:0/2213837317 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40ec064b60 con 0x7f40d80038e0 2023-10-18T16:07:23.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.682+0000 7f40f0ab5700 1 -- 172.21.15.88:0/2213837317 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40d800bfc0 con 0x7f40d8005050 2023-10-18T16:07:23.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.682+0000 7f40f0ab5700 1 -- 172.21.15.88:0/2213837317 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40d800c180 con 0x7f40d8004690 2023-10-18T16:07:23.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.682+0000 7f40f1ab7700 1 --2- 172.21.15.88:0/2213837317 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40d80038e0 0x7f40d8003ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:23.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.683+0000 7f40f22b8700 1 --2- 172.21.15.88:0/2213837317 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40d8005050 0x7f40d800b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:23.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.683+0000 7f40f12b6700 1 --2- 172.21.15.88:0/2213837317 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40d8004690 0x7f40d8004b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:23.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.683+0000 7f40f1ab7700 1 -- 172.21.15.88:0/2213837317 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40d8004690 msgr2=0x7f40d8004b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:23.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.683+0000 7f40f1ab7700 1 --2- 172.21.15.88:0/2213837317 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40d8004690 0x7f40d8004b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:23.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.683+0000 7f40f1ab7700 1 -- 172.21.15.88:0/2213837317 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40d8005050 msgr2=0x7f40d800b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:23.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.683+0000 7f40f1ab7700 1 --2- 172.21.15.88:0/2213837317 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40d8005050 0x7f40d800b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:23.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.683+0000 7f40f1ab7700 1 -- 172.21.15.88:0/2213837317 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40d80033f0 con 0x7f40d80038e0 2023-10-18T16:07:23.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.683+0000 7f40f1ab7700 1 --2- 172.21.15.88:0/2213837317 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40d80038e0 0x7f40d8003ce0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f40e8006db0 tx=0x7f40e8008040).ready entity=mon.0 client_cookie=e44e62959b1ebc09 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:23.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.683+0000 7f40f12b6700 1 --2- 172.21.15.88:0/2213837317 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40d8004690 0x7f40d8004b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-10-18T16:07:23.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.683+0000 7f40f0ab5700 1 -- 172.21.15.88:0/2213837317 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40e801d070 con 0x7f40d80038e0 2023-10-18T16:07:23.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.683+0000 7f40f0ab5700 1 -- 172.21.15.88:0/2213837317 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40e8015ca0 con 0x7f40d80038e0 2023-10-18T16:07:23.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.683+0000 7f40f0ab5700 1 -- 172.21.15.88:0/2213837317 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40e8022630 con 0x7f40d80038e0 2023-10-18T16:07:23.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.684+0000 7f40f2ab9700 1 -- 172.21.15.88:0/2213837317 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40d80038e0 msgr2=0x7f40d8003ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:23.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.684+0000 7f40f2ab9700 1 --2- 172.21.15.88:0/2213837317 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40d80038e0 0x7f40d8003ce0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f40e8006db0 tx=0x7f40e8008040).stop 2023-10-18T16:07:23.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.684+0000 7f40f2ab9700 1 -- 172.21.15.88:0/2213837317 shutdown_connections 2023-10-18T16:07:23.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.684+0000 7f40f2ab9700 1 --2- 172.21.15.88:0/2213837317 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40d8005050 0x7f40d800b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:23.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.684+0000 7f40f2ab9700 1 --2- 172.21.15.88:0/2213837317 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40d8004690 0x7f40d8004b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:23.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.684+0000 7f40f2ab9700 1 --2- 172.21.15.88:0/2213837317 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40d80038e0 0x7f40d8003ce0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:23.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.684+0000 7f40f2ab9700 1 -- 172.21.15.88:0/2213837317 >> 172.21.15.88:0/2213837317 conn(0x7f40ec05e050 msgr2=0x7f40ec05e460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:23.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.684+0000 7f40f2ab9700 1 -- 172.21.15.88:0/2213837317 shutdown_connections 2023-10-18T16:07:23.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.684+0000 7f40f2ab9700 1 -- 172.21.15.88:0/2213837317 wait complete. 2023-10-18T16:07:23.698 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.686+0000 7f40f2ab9700 1 Processor -- start 2023-10-18T16:07:23.698 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.686+0000 7f40f2ab9700 1 -- start start 2023-10-18T16:07:23.698 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.687+0000 7f40f2ab9700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40d80038e0 0x7f40ec14ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:23.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.687+0000 7f40f2ab9700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40d8004690 0x7f40ec163c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:23.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.687+0000 7f40f2ab9700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40d8005050 0x7f40ec147370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:23.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.687+0000 7f40f2ab9700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40ec0a4a60 con 0x7f40d80038e0 2023-10-18T16:07:23.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.687+0000 7f40f2ab9700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40ec1608a0 con 0x7f40d8004690 2023-10-18T16:07:23.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.687+0000 7f40f2ab9700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40ec14d330 con 0x7f40d8005050 2023-10-18T16:07:23.700 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.687+0000 7f40f1ab7700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40d80038e0 0x7f40ec14ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:23.700 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.687+0000 7f40f1ab7700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40d80038e0 0x7f40ec14ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56908/0 (socket says 172.21.15.88:56908) 2023-10-18T16:07:23.700 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.687+0000 7f40f1ab7700 1 -- 172.21.15.88:0/3480423313 learned_addr learned my addr 172.21.15.88:0/3480423313 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:23.700 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.687+0000 7f40f12b6700 1 --2- 172.21.15.88:0/3480423313 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40d8004690 0x7f40ec163c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:23.700 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.687+0000 7f40f22b8700 1 --2- 172.21.15.88:0/3480423313 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40d8005050 0x7f40ec147370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:23.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.687+0000 7f40f22b8700 1 -- 172.21.15.88:0/3480423313 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40d8005050 msgr2=0x7f40ec147370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2023-10-18T16:07:23.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.687+0000 7f40f22b8700 1 -- 172.21.15.88:0/3480423313 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40d8005050 msgr2=0x7f40ec147370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-18T16:07:23.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.687+0000 7f40f22b8700 1 --2- 172.21.15.88:0/3480423313 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40d8005050 0x7f40ec147370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-10-18T16:07:23.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.687+0000 7f40f22b8700 1 --2- 172.21.15.88:0/3480423313 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40d8005050 0x7f40ec147370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:07:23.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.687+0000 7f40f1ab7700 1 -- 172.21.15.88:0/3480423313 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40d8005050 msgr2=0x7f40ec147370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-18T16:07:23.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.687+0000 7f40f1ab7700 1 --2- 172.21.15.88:0/3480423313 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40d8005050 0x7f40ec147370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:23.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.687+0000 7f40f1ab7700 1 -- 172.21.15.88:0/3480423313 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40d8004690 msgr2=0x7f40ec163c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:23.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.687+0000 7f40f1ab7700 1 --2- 172.21.15.88:0/3480423313 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40d8004690 0x7f40ec163c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:23.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.687+0000 7f40f1ab7700 1 -- 172.21.15.88:0/3480423313 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40ec164170 con 0x7f40d80038e0 2023-10-18T16:07:23.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.687+0000 7f40f1ab7700 1 --2- 172.21.15.88:0/3480423313 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40d80038e0 0x7f40ec14ca80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f40e8018cc0 tx=0x7f40e8024000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:23.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.688+0000 7f40e1ffb700 1 -- 172.21.15.88:0/3480423313 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40e8015500 con 0x7f40d80038e0 2023-10-18T16:07:23.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.688+0000 7f40e1ffb700 1 -- 172.21.15.88:0/3480423313 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40e8017640 con 0x7f40d80038e0 2023-10-18T16:07:23.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.688+0000 7f40e1ffb700 1 -- 172.21.15.88:0/3480423313 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40e8017a00 con 0x7f40d80038e0 2023-10-18T16:07:23.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.688+0000 7f40f2ab9700 1 -- 172.21.15.88:0/3480423313 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40ec14d050 con 0x7f40d80038e0 2023-10-18T16:07:23.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.688+0000 7f40f2ab9700 1 -- 172.21.15.88:0/3480423313 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40ec1601e0 con 0x7f40d80038e0 2023-10-18T16:07:23.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.689+0000 7f40e1ffb700 1 -- 172.21.15.88:0/3480423313 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 1) v1 ==== 495+0+0 (secure 0 0 0) 0x7f40e802e420 con 0x7f40d80038e0 2023-10-18T16:07:23.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.689+0000 7f40e1ffb700 1 -- 172.21.15.88:0/3480423313 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f40e8015e80 con 0x7f40d80038e0 2023-10-18T16:07:23.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.689+0000 7f40f2ab9700 1 -- 172.21.15.88:0/3480423313 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40ec084180 con 0x7f40d80038e0 2023-10-18T16:07:23.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.693+0000 7f40e1ffb700 1 -- 172.21.15.88:0/3480423313 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73265 (secure 0 0 0) 0x7f40e804de50 con 0x7f40d80038e0 2023-10-18T16:07:23.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.765+0000 7f40f2ab9700 1 -- 172.21.15.88:0/3480423313 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f40ec084180 con 0x7f40d80038e0 2023-10-18T16:07:23.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.770+0000 7f40e1ffb700 1 -- 172.21.15.88:0/3480423313 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f40e803c020 con 0x7f40d80038e0 2023-10-18T16:07:23.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.771+0000 7f40f2ab9700 1 -- 172.21.15.88:0/3480423313 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40d80038e0 msgr2=0x7f40ec14ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:23.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.771+0000 7f40f2ab9700 1 --2- 172.21.15.88:0/3480423313 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40d80038e0 0x7f40ec14ca80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f40e8018cc0 tx=0x7f40e8024000).stop 2023-10-18T16:07:23.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.771+0000 7f40f2ab9700 1 -- 172.21.15.88:0/3480423313 shutdown_connections 2023-10-18T16:07:23.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.771+0000 7f40f2ab9700 1 --2- 172.21.15.88:0/3480423313 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40d8005050 0x7f40ec147370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:23.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.771+0000 7f40f2ab9700 1 --2- 172.21.15.88:0/3480423313 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40d80038e0 0x7f40ec14ca80 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:23.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.771+0000 7f40f2ab9700 1 --2- 172.21.15.88:0/3480423313 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40d8004690 0x7f40ec163c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:23.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.771+0000 7f40f2ab9700 1 -- 172.21.15.88:0/3480423313 >> 172.21.15.88:0/3480423313 conn(0x7f40ec05e050 msgr2=0x7f40ec0c1090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:23.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.771+0000 7f40f2ab9700 1 -- 172.21.15.88:0/3480423313 shutdown_connections 2023-10-18T16:07:23.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:23.771+0000 7f40f2ab9700 1 -- 172.21.15.88:0/3480423313 wait complete. 2023-10-18T16:07:23.777 INFO:teuthology.orchestra.run.smithi088.stderr:adjusted tunables profile to default 2023-10-18T16:07:23.787 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-10-18T16:07:23.787 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-18T16:07:23.788 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-10-18T16:07:23.816 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-18T16:07:23.817 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-10-18T16:07:23.832 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-10-18T16:07:23.828+0000 7f07f6041680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-18T16:07:23.833 INFO:tasks.ceph.mgr.x.smithi151.stderr:2023-10-18T16:07:23.829+0000 7f8d9db54680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-18T16:07:23.857 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-18T16:07:23.857 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-10-18T16:07:23.901 INFO:tasks.ceph.mgr.x.smithi151.stderr:2023-10-18T16:07:23.897+0000 7f8d9db54680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-18T16:07:23.905 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-10-18T16:07:23.902+0000 7f07f6041680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-18T16:07:23.921 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-18T16:07:23.921 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-10-18T16:07:23.987 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-10-18T16:07:23.987 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-10-18T16:07:24.012 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-10-18T16:07:24.012 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-10-18T16:07:24.077 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-10-18T16:07:24.078 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-10-18T16:07:24.143 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-10-18T16:07:24.143 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-10-18T16:07:24.209 DEBUG:teuthology.orchestra.run.smithi151:> sudo ceph --cluster ceph osd new e421c20c-f7bc-4574-97a2-201bb4a16b4b 0 2023-10-18T16:07:24.361 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.357+0000 7f9533027700 1 Processor -- start 2023-10-18T16:07:24.361 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.357+0000 7f9533027700 1 -- start start 2023-10-18T16:07:24.364 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.358+0000 7f9533027700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f952c0bbb50 0x7f952c0c1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:24.364 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.358+0000 7f9533027700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f952c0bbf30 0x7f952c150bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:24.364 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.358+0000 7f9533027700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f952c0c3030 con 0x7f952c0c71b0 2023-10-18T16:07:24.364 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.358+0000 7f9533027700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f952c1510b0 con 0x7f952c0bbb50 2023-10-18T16:07:24.365 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.358+0000 7f9533027700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f952c0c2140 con 0x7f952c0bbf30 2023-10-18T16:07:24.365 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.358+0000 7f952bfff700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f952c0c71b0 0x7f952c14d2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:58178/0 (socket says 172.21.15.151:58178) 2023-10-18T16:07:24.365 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.358+0000 7f95315c4700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f952c0bbf30 0x7f952c150bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:24.365 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.358+0000 7f952bfff700 1 -- 172.21.15.151:0/3527069547 learned_addr learned my addr 172.21.15.151:0/3527069547 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-10-18T16:07:24.365 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.358+0000 7f9530dc3700 1 --2- 172.21.15.151:0/3527069547 >> v2:172.21.15.88:3300/0 conn(0x7f952c0bbb50 0x7f952c0c1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:24.366 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.358+0000 7f95315c4700 1 -- 172.21.15.151:0/3527069547 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f952c0bbf30 msgr2=0x7f952c150bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-18T16:07:24.366 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.358+0000 7f95315c4700 1 -- 172.21.15.151:0/3527069547 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f952c0bbf30 msgr2=0x7f952c150bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-18T16:07:24.366 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.358+0000 7f95315c4700 1 --2- 172.21.15.151:0/3527069547 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f952c0bbf30 0x7f952c150bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-10-18T16:07:24.366 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.358+0000 7f95315c4700 1 --2- 172.21.15.151:0/3527069547 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f952c0bbf30 0x7f952c150bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:07:24.366 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.358+0000 7f952b7fe700 1 -- 172.21.15.151:0/3527069547 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3105771349 0 0) 0x7f952c0c3030 con 0x7f952c0c71b0 2023-10-18T16:07:24.367 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.358+0000 7f952b7fe700 1 -- 172.21.15.151:0/3527069547 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f951c0033f0 con 0x7f952c0c71b0 2023-10-18T16:07:24.367 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.359+0000 7f9530dc3700 1 -- 172.21.15.151:0/3527069547 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f952c0bbf30 msgr2=0x7f952c150bc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:07:24.367 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.359+0000 7f9530dc3700 1 --2- 172.21.15.151:0/3527069547 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f952c0bbf30 0x7f952c150bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:24.367 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.359+0000 7f9530dc3700 1 -- 172.21.15.151:0/3527069547 >> v1:172.21.15.151:6789/0 conn(0x7f952c0c71b0 legacy=0x7f952c14d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:24.367 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.359+0000 7f9530dc3700 1 -- 172.21.15.151:0/3527069547 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f952c0c2420 con 0x7f952c0bbb50 2023-10-18T16:07:24.368 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.359+0000 7f9530dc3700 1 --2- 172.21.15.151:0/3527069547 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f952c0bbb50 0x7f952c0c1bb0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f9520009760 tx=0x7f9520009060).ready entity=mon.0 client_cookie=8b7d39ab66b7c8b3 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:24.368 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.359+0000 7f952b7fe700 1 -- 172.21.15.151:0/3527069547 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f952000b590 con 0x7f952c0bbb50 2023-10-18T16:07:24.368 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.359+0000 7f952b7fe700 1 -- 172.21.15.151:0/3527069547 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f952000b6f0 con 0x7f952c0bbb50 2023-10-18T16:07:24.368 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.359+0000 7f952b7fe700 1 -- 172.21.15.151:0/3527069547 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f952000ba70 con 0x7f952c0bbb50 2023-10-18T16:07:24.368 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.359+0000 7f9533027700 1 -- 172.21.15.151:0/3527069547 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f952c0bbb50 msgr2=0x7f952c0c1bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:24.369 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.359+0000 7f9533027700 1 --2- 172.21.15.151:0/3527069547 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f952c0bbb50 0x7f952c0c1bb0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f9520009760 tx=0x7f9520009060).stop 2023-10-18T16:07:24.369 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.359+0000 7f9533027700 1 -- 172.21.15.151:0/3527069547 shutdown_connections 2023-10-18T16:07:24.369 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.359+0000 7f9533027700 1 --2- 172.21.15.151:0/3527069547 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f952c0bbf30 0x7f952c150bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:24.369 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.359+0000 7f9533027700 1 --2- 172.21.15.151:0/3527069547 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f952c0bbb50 0x7f952c0c1bb0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:24.369 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.359+0000 7f9533027700 1 -- 172.21.15.151:0/3527069547 >> 172.21.15.151:0/3527069547 conn(0x7f952c06a810 msgr2=0x7f952c0c6a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:24.370 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.359+0000 7f9533027700 1 -- 172.21.15.151:0/3527069547 shutdown_connections 2023-10-18T16:07:24.370 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.360+0000 7f9533027700 1 -- 172.21.15.151:0/3527069547 wait complete. 2023-10-18T16:07:24.370 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.361+0000 7f9533027700 1 Processor -- start 2023-10-18T16:07:24.373 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.361+0000 7f9533027700 1 -- start start 2023-10-18T16:07:24.373 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.361+0000 7f9533027700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f952c0bbb50 0x7f952c14e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:24.373 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.361+0000 7f9530dc3700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f952c0bbb50 0x7f952c14e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:24.373 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.361+0000 7f9530dc3700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f952c0bbb50 0x7f952c14e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:58404/0 (socket says 172.21.15.151:58404) 2023-10-18T16:07:24.374 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.362+0000 7f9533027700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f952c0bbf30 0x7f952c143380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:24.374 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.362+0000 7f9530dc3700 1 -- 172.21.15.151:0/426079924 learned_addr learned my addr 172.21.15.151:0/426079924 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-10-18T16:07:24.374 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.362+0000 7f9533027700 1 --2- 172.21.15.151:0/426079924 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f952c0c71b0 0x7f952c143870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:24.375 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.362+0000 7f9533027700 1 -- 172.21.15.151:0/426079924 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f952c14fb60 con 0x7f952c0bbf30 2023-10-18T16:07:24.375 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.362+0000 7f9533027700 1 -- 172.21.15.151:0/426079924 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f952c14d7f0 con 0x7f952c0c71b0 2023-10-18T16:07:24.375 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.362+0000 7f9533027700 1 -- 172.21.15.151:0/426079924 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f952c14ddb0 con 0x7f952c0bbb50 2023-10-18T16:07:24.375 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.362+0000 7f95315c4700 1 --2- 172.21.15.151:0/426079924 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f952c0c71b0 0x7f952c143870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:24.375 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.362+0000 7f9530dc3700 1 -- 172.21.15.151:0/426079924 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f952c0bbb50 msgr2=0x7f952c14e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2023-10-18T16:07:24.376 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.362+0000 7f9530dc3700 1 -- 172.21.15.151:0/426079924 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f952c0bbb50 msgr2=0x7f952c14e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-18T16:07:24.376 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.362+0000 7f9530dc3700 1 --2- 172.21.15.151:0/426079924 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f952c0bbb50 0x7f952c14e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-10-18T16:07:24.376 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.362+0000 7f952bfff700 1 --2- 172.21.15.151:0/426079924 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f952c0bbf30 0x7f952c143380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:24.376 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.362+0000 7f9530dc3700 1 --2- 172.21.15.151:0/426079924 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f952c0bbb50 0x7f952c14e240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:07:24.376 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.362+0000 7f95315c4700 1 -- 172.21.15.151:0/426079924 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f952c0bbb50 msgr2=0x7f952c14e240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-18T16:07:24.376 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.362+0000 7f95315c4700 1 --2- 172.21.15.151:0/426079924 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f952c0bbb50 0x7f952c14e240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:24.377 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.362+0000 7f95315c4700 1 -- 172.21.15.151:0/426079924 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f952c0bbf30 msgr2=0x7f952c143380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:24.377 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.362+0000 7f95315c4700 1 --2- 172.21.15.151:0/426079924 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f952c0bbf30 0x7f952c143380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:24.377 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.362+0000 7f95315c4700 1 -- 172.21.15.151:0/426079924 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95200092a0 con 0x7f952c0c71b0 2023-10-18T16:07:24.377 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.362+0000 7f952bfff700 1 --2- 172.21.15.151:0/426079924 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f952c0bbf30 0x7f952c143380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2023-10-18T16:07:24.377 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.362+0000 7f95315c4700 1 --2- 172.21.15.151:0/426079924 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f952c0c71b0 0x7f952c143870 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f9524007ea0 tx=0x7f95240097e0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:24.378 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.362+0000 7f9528ff9700 1 -- 172.21.15.151:0/426079924 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f952400eb70 con 0x7f952c0c71b0 2023-10-18T16:07:24.378 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.362+0000 7f9528ff9700 1 -- 172.21.15.151:0/426079924 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f952400ecd0 con 0x7f952c0c71b0 2023-10-18T16:07:24.378 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.362+0000 7f9528ff9700 1 -- 172.21.15.151:0/426079924 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9524014610 con 0x7f952c0c71b0 2023-10-18T16:07:24.378 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.362+0000 7f9533027700 1 -- 172.21.15.151:0/426079924 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f952c161020 con 0x7f952c0c71b0 2023-10-18T16:07:24.378 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.362+0000 7f9533027700 1 -- 172.21.15.151:0/426079924 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f952c0aa820 con 0x7f952c0c71b0 2023-10-18T16:07:24.379 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.363+0000 7f9528ff9700 1 -- 172.21.15.151:0/426079924 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 1) v1 ==== 495+0+0 (secure 0 0 0) 0x7f9524014ca0 con 0x7f952c0c71b0 2023-10-18T16:07:24.379 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.363+0000 7f9528ff9700 1 -- 172.21.15.151:0/426079924 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 754+0+0 (secure 0 0 0) 0x7f952401e8c0 con 0x7f952c0c71b0 2023-10-18T16:07:24.379 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.363+0000 7f9533027700 1 -- 172.21.15.151:0/426079924 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f952c06c4e0 con 0x7f952c0c71b0 2023-10-18T16:07:24.379 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.366+0000 7f9528ff9700 1 -- 172.21.15.151:0/426079924 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73265 (secure 0 0 0) 0x7f9524023060 con 0x7f952c0c71b0 2023-10-18T16:07:24.387 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-10-18T16:07:24.384+0000 7f07f6041680 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-18T16:07:24.420 INFO:tasks.ceph.mgr.x.smithi151.stderr:2023-10-18T16:07:24.416+0000 7f8d9db54680 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-18T16:07:24.449 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.445+0000 7f9533027700 1 -- 172.21.15.151:0/426079924 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd new", "uuid": "e421c20c-f7bc-4574-97a2-201bb4a16b4b", "id": 0} v 0) v1 -- 0x7f952c0c2700 con 0x7f952c0c71b0 2023-10-18T16:07:24.455 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.451+0000 7f9528ff9700 1 -- 172.21.15.151:0/426079924 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e421c20c-f7bc-4574-97a2-201bb4a16b4b", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f9524019070 con 0x7f952c0c71b0 2023-10-18T16:07:24.455 INFO:teuthology.orchestra.run.smithi151.stdout:0 2023-10-18T16:07:24.457 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.452+0000 7f9533027700 1 -- 172.21.15.151:0/426079924 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f952c0c71b0 msgr2=0x7f952c143870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:24.457 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.452+0000 7f9533027700 1 --2- 172.21.15.151:0/426079924 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f952c0c71b0 0x7f952c143870 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f9524007ea0 tx=0x7f95240097e0).stop 2023-10-18T16:07:24.457 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.453+0000 7f9533027700 1 -- 172.21.15.151:0/426079924 shutdown_connections 2023-10-18T16:07:24.457 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.453+0000 7f9533027700 1 --2- 172.21.15.151:0/426079924 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f952c0c71b0 0x7f952c143870 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:24.457 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.453+0000 7f9533027700 1 --2- 172.21.15.151:0/426079924 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f952c0bbb50 0x7f952c14e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:24.457 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.453+0000 7f9533027700 1 --2- 172.21.15.151:0/426079924 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f952c0bbf30 0x7f952c143380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:24.458 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.453+0000 7f9533027700 1 -- 172.21.15.151:0/426079924 >> 172.21.15.151:0/426079924 conn(0x7f952c06a810 msgr2=0x7f952c0c1080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:24.458 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.453+0000 7f9533027700 1 -- 172.21.15.151:0/426079924 shutdown_connections 2023-10-18T16:07:24.458 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.453+0000 7f9533027700 1 -- 172.21.15.151:0/426079924 wait complete. 2023-10-18T16:07:24.468 DEBUG:teuthology.orchestra.run.smithi151:> sudo ceph --cluster ceph osd new 9485c5ec-379a-4038-a2d8-1f1081fe2eea 1 2023-10-18T16:07:24.580 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.576+0000 7fdfdf4b0700 1 Processor -- start 2023-10-18T16:07:24.580 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.576+0000 7fdfdf4b0700 1 -- start start 2023-10-18T16:07:24.587 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.577+0000 7fdfdf4b0700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdfd80c71b0 0x7fdfd80c1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:24.587 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.577+0000 7fdfdf4b0700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfd80c3030 con 0x7fdfd80bbb50 2023-10-18T16:07:24.587 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.577+0000 7fdfdf4b0700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfd80c2140 con 0x7fdfd80bbf30 2023-10-18T16:07:24.587 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.577+0000 7fdfdf4b0700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd80c2420 con 0x7fdfd80c71b0 2023-10-18T16:07:24.588 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.577+0000 7fdfdca4b700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdfd80c71b0 0x7fdfd80c1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:24.588 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.577+0000 7fdfdca4b700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdfd80c71b0 0x7fdfd80c1bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:58412/0 (socket says 172.21.15.151:58412) 2023-10-18T16:07:24.588 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.577+0000 7fdfdda4d700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fdfd80bbf30 0x7fdfd8063a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:58192/0 (socket says 172.21.15.151:58192) 2023-10-18T16:07:24.588 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.577+0000 7fdfdca4b700 1 -- 172.21.15.151:0/202718613 learned_addr learned my addr 172.21.15.151:0/202718613 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-10-18T16:07:24.588 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.577+0000 7fdfdca4b700 1 -- 172.21.15.151:0/202718613 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdfd80c71b0 msgr2=0x7fdfd80c1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-18T16:07:24.589 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.577+0000 7fdfdca4b700 1 -- 172.21.15.151:0/202718613 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdfd80c71b0 msgr2=0x7fdfd80c1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-18T16:07:24.589 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.577+0000 7fdfdca4b700 1 --2- 172.21.15.151:0/202718613 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdfd80c71b0 0x7fdfd80c1bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-10-18T16:07:24.589 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.577+0000 7fdfdca4b700 1 --2- 172.21.15.151:0/202718613 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdfd80c71b0 0x7fdfd80c1bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:07:24.589 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.577+0000 7fdfcffff700 1 -- 172.21.15.151:0/202718613 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 943415450 0 0) 0x7fdfd80c2140 con 0x7fdfd80bbf30 2023-10-18T16:07:24.589 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.577+0000 7fdfcffff700 1 -- 172.21.15.151:0/202718613 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfc40033f0 con 0x7fdfd80bbf30 2023-10-18T16:07:24.589 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.578+0000 7fdfcffff700 1 -- 172.21.15.151:0/202718613 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 828031202 0 0) 0x7fdfd80c3030 con 0x7fdfd80bbb50 2023-10-18T16:07:24.590 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.578+0000 7fdfcffff700 1 -- 172.21.15.151:0/202718613 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfd80c2140 con 0x7fdfd80bbb50 2023-10-18T16:07:24.590 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.578+0000 7fdfcffff700 1 -- 172.21.15.151:0/202718613 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2184818832 0 0) 0x7fdfc40033f0 con 0x7fdfd80bbf30 2023-10-18T16:07:24.590 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.578+0000 7fdfcffff700 1 -- 172.21.15.151:0/202718613 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdfd80c71b0 msgr2=0x7fdfd80c1bb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:07:24.590 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.578+0000 7fdfcffff700 1 --2- 172.21.15.151:0/202718613 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdfd80c71b0 0x7fdfd80c1bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:24.590 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.578+0000 7fdfcffff700 1 -- 172.21.15.151:0/202718613 >> v1:172.21.15.88:6789/0 conn(0x7fdfd80bbb50 legacy=0x7fdfd814d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:24.591 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.578+0000 7fdfcffff700 1 -- 172.21.15.151:0/202718613 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfd80c2700 con 0x7fdfd80bbf30 2023-10-18T16:07:24.591 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.578+0000 7fdfcffff700 1 -- 172.21.15.151:0/202718613 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441715075 0 0) 0x7fdfd00017f0 con 0x7fdfd80bbf30 2023-10-18T16:07:24.591 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.578+0000 7fdfcffff700 1 -- 172.21.15.151:0/202718613 >> v1:172.21.15.151:6789/0 conn(0x7fdfd80bbf30 legacy=0x7fdfd8063a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:24.591 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.578+0000 7fdfcffff700 1 --2- 172.21.15.151:0/202718613 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdfc4003a70 0x7fdfc4003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:24.592 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.578+0000 7fdfcffff700 1 --2- 172.21.15.151:0/202718613 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdfd80c71b0 0x7fdfc4004690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:24.592 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.578+0000 7fdfcffff700 1 --2- 172.21.15.151:0/202718613 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdfc4004b80 0x7fdfc4005000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:24.592 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.578+0000 7fdfcffff700 1 -- 172.21.15.151:0/202718613 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd80c2420 con 0x7fdfc4004b80 2023-10-18T16:07:24.592 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.578+0000 7fdfcffff700 1 -- 172.21.15.151:0/202718613 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfc400b990 con 0x7fdfc4003a70 2023-10-18T16:07:24.592 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.578+0000 7fdfcffff700 1 -- 172.21.15.151:0/202718613 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfc400bb50 con 0x7fdfd80c71b0 2023-10-18T16:07:24.593 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.578+0000 7fdfdd24c700 1 --2- 172.21.15.151:0/202718613 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdfc4003a70 0x7fdfc4003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:24.593 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.578+0000 7fdfdca4b700 1 --2- 172.21.15.151:0/202718613 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdfd80c71b0 0x7fdfc4004690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:24.593 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.578+0000 7fdfdca4b700 1 -- 172.21.15.151:0/202718613 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdfd80c71b0 msgr2=0x7fdfc4004690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-18T16:07:24.593 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.578+0000 7fdfdca4b700 1 -- 172.21.15.151:0/202718613 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdfd80c71b0 msgr2=0x7fdfc4004690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-18T16:07:24.593 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.578+0000 7fdfdca4b700 1 --2- 172.21.15.151:0/202718613 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdfd80c71b0 0x7fdfc4004690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-10-18T16:07:24.594 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.578+0000 7fdfdca4b700 1 --2- 172.21.15.151:0/202718613 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdfd80c71b0 0x7fdfc4004690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:07:24.594 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.578+0000 7fdfdda4d700 1 --2- 172.21.15.151:0/202718613 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdfc4004b80 0x7fdfc4005000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:24.594 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.578+0000 7fdfdd24c700 1 -- 172.21.15.151:0/202718613 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdfd80c71b0 msgr2=0x7fdfc4004690 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:07:24.594 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.578+0000 7fdfdd24c700 1 --2- 172.21.15.151:0/202718613 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdfd80c71b0 0x7fdfc4004690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:24.594 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.578+0000 7fdfdd24c700 1 -- 172.21.15.151:0/202718613 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdfc4004b80 msgr2=0x7fdfc4005000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:24.595 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.578+0000 7fdfdd24c700 1 --2- 172.21.15.151:0/202718613 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdfc4004b80 0x7fdfc4005000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:24.595 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.578+0000 7fdfdd24c700 1 -- 172.21.15.151:0/202718613 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfc40033f0 con 0x7fdfc4003a70 2023-10-18T16:07:24.595 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.578+0000 7fdfdd24c700 1 --2- 172.21.15.151:0/202718613 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdfc4003a70 0x7fdfc4003e70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fdfc8001890 tx=0x7fdfc8007bb0).ready entity=mon.1 client_cookie=4c0c357b3ae6bb28 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:24.595 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.578+0000 7fdfcffff700 1 -- 172.21.15.151:0/202718613 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfc8007020 con 0x7fdfc4003a70 2023-10-18T16:07:24.595 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.578+0000 7fdfcffff700 1 -- 172.21.15.151:0/202718613 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfc8007180 con 0x7fdfc4003a70 2023-10-18T16:07:24.596 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.578+0000 7fdfcffff700 1 -- 172.21.15.151:0/202718613 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfc8007500 con 0x7fdfc4003a70 2023-10-18T16:07:24.596 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.579+0000 7fdfdf4b0700 1 -- 172.21.15.151:0/202718613 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdfc4003a70 msgr2=0x7fdfc4003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:24.596 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.579+0000 7fdfdf4b0700 1 --2- 172.21.15.151:0/202718613 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdfc4003a70 0x7fdfc4003e70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fdfc8001890 tx=0x7fdfc8007bb0).stop 2023-10-18T16:07:24.596 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.579+0000 7fdfdda4d700 1 -- 172.21.15.151:0/202718613 reap_dead start 2023-10-18T16:07:24.596 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.579+0000 7fdfdf4b0700 1 -- 172.21.15.151:0/202718613 shutdown_connections 2023-10-18T16:07:24.597 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.579+0000 7fdfdf4b0700 1 -- 172.21.15.151:0/202718613 >> 172.21.15.151:0/202718613 conn(0x7fdfd806a810 msgr2=0x7fdfd80c6a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:24.597 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.579+0000 7fdfdf4b0700 1 -- 172.21.15.151:0/202718613 shutdown_connections 2023-10-18T16:07:24.597 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.579+0000 7fdfdf4b0700 1 -- 172.21.15.151:0/202718613 wait complete. 2023-10-18T16:07:24.597 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.580+0000 7fdfdf4b0700 1 Processor -- start 2023-10-18T16:07:24.597 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.580+0000 7fdfdf4b0700 1 -- start start 2023-10-18T16:07:24.597 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.581+0000 7fdfdf4b0700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdfc4003a70 0x7fdfd80c0550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:24.598 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.581+0000 7fdfdf4b0700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdfd80c0aa0 0x7fdfd814f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:24.598 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.581+0000 7fdfdf4b0700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdfd814f610 0x7fdfd8160610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:24.598 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.581+0000 7fdfdf4b0700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd81601f0 con 0x7fdfc4003a70 2023-10-18T16:07:24.598 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.581+0000 7fdfdf4b0700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd815ff00 con 0x7fdfd814f610 2023-10-18T16:07:24.599 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.581+0000 7fdfdf4b0700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd80ac550 con 0x7fdfd80c0aa0 2023-10-18T16:07:24.599 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.581+0000 7fdfdca4b700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdfd80c0aa0 0x7fdfd814f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:24.599 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.581+0000 7fdfdda4d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdfd814f610 0x7fdfd8160610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:24.599 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.581+0000 7fdfdda4d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdfd814f610 0x7fdfd8160610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:57260/0 (socket says 172.21.15.151:57260) 2023-10-18T16:07:24.600 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.581+0000 7fdfdca4b700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdfd80c0aa0 0x7fdfd814f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:58424/0 (socket says 172.21.15.151:58424) 2023-10-18T16:07:24.600 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.581+0000 7fdfdd24c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdfc4003a70 0x7fdfd80c0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:24.600 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.581+0000 7fdfdda4d700 1 -- 172.21.15.151:0/290318175 learned_addr learned my addr 172.21.15.151:0/290318175 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-10-18T16:07:24.600 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.581+0000 7fdfdca4b700 1 -- 172.21.15.151:0/290318175 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdfd80c0aa0 msgr2=0x7fdfd814f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-10-18T16:07:24.600 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.581+0000 7fdfdca4b700 1 -- 172.21.15.151:0/290318175 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdfd80c0aa0 msgr2=0x7fdfd814f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-18T16:07:24.600 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.581+0000 7fdfdca4b700 1 --2- 172.21.15.151:0/290318175 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdfd80c0aa0 0x7fdfd814f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-10-18T16:07:24.601 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.581+0000 7fdfdca4b700 1 --2- 172.21.15.151:0/290318175 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdfd80c0aa0 0x7fdfd814f120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:07:24.601 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.581+0000 7fdfdda4d700 1 -- 172.21.15.151:0/290318175 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdfd80c0aa0 msgr2=0x7fdfd814f120 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-18T16:07:24.601 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.581+0000 7fdfdda4d700 1 --2- 172.21.15.151:0/290318175 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdfd80c0aa0 0x7fdfd814f120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:24.601 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.581+0000 7fdfdda4d700 1 -- 172.21.15.151:0/290318175 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdfc4003a70 msgr2=0x7fdfd80c0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:24.601 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.581+0000 7fdfdda4d700 1 --2- 172.21.15.151:0/290318175 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdfc4003a70 0x7fdfd80c0550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:24.602 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.581+0000 7fdfdda4d700 1 -- 172.21.15.151:0/290318175 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfd80aa030 con 0x7fdfd814f610 2023-10-18T16:07:24.602 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.581+0000 7fdfdda4d700 1 --2- 172.21.15.151:0/290318175 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdfd814f610 0x7fdfd8160610 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7fdfd0001350 tx=0x7fdfd0002170).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:24.602 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.582+0000 7fdfcd7fa700 1 -- 172.21.15.151:0/290318175 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd000f8a0 con 0x7fdfd814f610 2023-10-18T16:07:24.603 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.582+0000 7fdfcd7fa700 1 -- 172.21.15.151:0/290318175 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfd000fa00 con 0x7fdfd814f610 2023-10-18T16:07:24.603 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.582+0000 7fdfcd7fa700 1 -- 172.21.15.151:0/290318175 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd000fdc0 con 0x7fdfd814f610 2023-10-18T16:07:24.603 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.582+0000 7fdfdf4b0700 1 -- 172.21.15.151:0/290318175 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfd80ac7a0 con 0x7fdfd814f610 2023-10-18T16:07:24.603 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.582+0000 7fdfdf4b0700 1 -- 172.21.15.151:0/290318175 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfd8150070 con 0x7fdfd814f610 2023-10-18T16:07:24.604 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.582+0000 7fdfcd7fa700 1 -- 172.21.15.151:0/290318175 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 1) v1 ==== 495+0+0 (secure 0 0 0) 0x7fdfd00127c0 con 0x7fdfd814f610 2023-10-18T16:07:24.604 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.582+0000 7fdfcd7fa700 1 -- 172.21.15.151:0/290318175 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 869+0+0 (secure 0 0 0) 0x7fdfd001e440 con 0x7fdfd814f610 2023-10-18T16:07:24.604 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.583+0000 7fdfdf4b0700 1 -- 172.21.15.151:0/290318175 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfd8160460 con 0x7fdfd814f610 2023-10-18T16:07:24.605 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.585+0000 7fdfcd7fa700 1 -- 172.21.15.151:0/290318175 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73265 (secure 0 0 0) 0x7fdfd0019070 con 0x7fdfd814f610 2023-10-18T16:07:24.667 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.663+0000 7fdfdf4b0700 1 -- 172.21.15.151:0/290318175 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd new", "uuid": "9485c5ec-379a-4038-a2d8-1f1081fe2eea", "id": 1} v 0) v1 -- 0x7fdfd806c4e0 con 0x7fdfd814f610 2023-10-18T16:07:24.672 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.668+0000 7fdfcd7fa700 1 -- 172.21.15.151:0/290318175 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9485c5ec-379a-4038-a2d8-1f1081fe2eea", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7fdfd0012c50 con 0x7fdfd814f610 2023-10-18T16:07:24.672 INFO:teuthology.orchestra.run.smithi151.stdout:1 2023-10-18T16:07:24.673 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.669+0000 7fdfdf4b0700 1 -- 172.21.15.151:0/290318175 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdfd814f610 msgr2=0x7fdfd8160610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:24.674 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.669+0000 7fdfdf4b0700 1 --2- 172.21.15.151:0/290318175 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdfd814f610 0x7fdfd8160610 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7fdfd0001350 tx=0x7fdfd0002170).stop 2023-10-18T16:07:24.674 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.669+0000 7fdfdf4b0700 1 -- 172.21.15.151:0/290318175 shutdown_connections 2023-10-18T16:07:24.674 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.669+0000 7fdfdf4b0700 1 --2- 172.21.15.151:0/290318175 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdfd814f610 0x7fdfd8160610 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:24.674 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.669+0000 7fdfdf4b0700 1 --2- 172.21.15.151:0/290318175 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdfc4003a70 0x7fdfd80c0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:24.674 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.669+0000 7fdfdf4b0700 1 --2- 172.21.15.151:0/290318175 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdfd80c0aa0 0x7fdfd814f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:24.675 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.669+0000 7fdfdf4b0700 1 -- 172.21.15.151:0/290318175 >> 172.21.15.151:0/290318175 conn(0x7fdfd806a810 msgr2=0x7fdfd8150b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:24.675 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.669+0000 7fdfdf4b0700 1 -- 172.21.15.151:0/290318175 shutdown_connections 2023-10-18T16:07:24.675 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.669+0000 7fdfdf4b0700 1 -- 172.21.15.151:0/290318175 wait complete. 2023-10-18T16:07:24.686 DEBUG:teuthology.orchestra.run.smithi151:> sudo ceph --cluster ceph osd new 671895e2-57f3-4f1a-ba4a-a580af1f3ec2 2 2023-10-18T16:07:24.797 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.793+0000 7f35d6798700 1 Processor -- start 2023-10-18T16:07:24.798 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.793+0000 7f35d6798700 1 -- start start 2023-10-18T16:07:24.800 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.795+0000 7f35d6798700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f35d0063050 0x7f35d00b6870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:24.801 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.795+0000 7f35d6798700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f35d0062650 0x7f35d014d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:24.801 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.795+0000 7f35d6798700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35d0066cc0 con 0x7f35d00b6490 2023-10-18T16:07:24.801 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.795+0000 7f35d6798700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35d0066fa0 con 0x7f35d0062650 2023-10-18T16:07:24.801 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.795+0000 7f35d6798700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35d014d940 con 0x7f35d0063050 2023-10-18T16:07:24.801 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.795+0000 7f35d4d35700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f35d0062650 0x7f35d014d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:24.802 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.795+0000 7f35cffff700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f35d00b6490 0x7f35d0067430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:58206/0 (socket says 172.21.15.151:58206) 2023-10-18T16:07:24.802 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.795+0000 7f35cffff700 1 -- 172.21.15.151:0/653450009 learned_addr learned my addr 172.21.15.151:0/653450009 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-10-18T16:07:24.802 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.795+0000 7f35cf7fe700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f35d0063050 0x7f35d00b6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:24.802 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.795+0000 7f35cf7fe700 1 -- 172.21.15.151:0/653450009 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f35d0063050 msgr2=0x7f35d00b6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-18T16:07:24.803 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.795+0000 7f35cf7fe700 1 -- 172.21.15.151:0/653450009 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f35d0063050 msgr2=0x7f35d00b6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-18T16:07:24.803 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.795+0000 7f35cf7fe700 1 --2- 172.21.15.151:0/653450009 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f35d0063050 0x7f35d00b6870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-10-18T16:07:24.803 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.795+0000 7f35cf7fe700 1 --2- 172.21.15.151:0/653450009 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f35d0063050 0x7f35d00b6870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:07:24.803 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.795+0000 7f35d4d35700 1 -- 172.21.15.151:0/653450009 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f35d0063050 msgr2=0x7f35d00b6870 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:07:24.803 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.795+0000 7f35d4d35700 1 --2- 172.21.15.151:0/653450009 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f35d0063050 0x7f35d00b6870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:24.804 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.795+0000 7f35d4d35700 1 -- 172.21.15.151:0/653450009 >> v1:172.21.15.151:6789/0 conn(0x7f35d00b6490 legacy=0x7f35d0067430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:24.804 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.795+0000 7f35ceffd700 1 -- 172.21.15.151:0/653450009 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 883104219 0 0) 0x7f35d0066cc0 con 0x7f35d00b6490 2023-10-18T16:07:24.804 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.795+0000 7f35d4d35700 1 -- 172.21.15.151:0/653450009 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35d0067b40 con 0x7f35d0062650 2023-10-18T16:07:24.804 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.795+0000 7f35d4d35700 1 --2- 172.21.15.151:0/653450009 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f35d0062650 0x7f35d014d450 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f35c8009e60 tx=0x7f35c8009780).ready entity=mon.1 client_cookie=e8d5410a6df0ed80 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:24.804 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.795+0000 7f35ceffd700 1 -- 172.21.15.151:0/653450009 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35c800e070 con 0x7f35d0062650 2023-10-18T16:07:24.805 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.796+0000 7f35ceffd700 1 -- 172.21.15.151:0/653450009 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35c800bcc0 con 0x7f35d0062650 2023-10-18T16:07:24.805 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.796+0000 7f35ceffd700 1 -- 172.21.15.151:0/653450009 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35c8012620 con 0x7f35d0062650 2023-10-18T16:07:24.805 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.796+0000 7f35d6798700 1 -- 172.21.15.151:0/653450009 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f35d0062650 msgr2=0x7f35d014d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:24.805 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.796+0000 7f35d6798700 1 --2- 172.21.15.151:0/653450009 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f35d0062650 0x7f35d014d450 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f35c8009e60 tx=0x7f35c8009780).stop 2023-10-18T16:07:24.805 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.796+0000 7f35d6798700 1 -- 172.21.15.151:0/653450009 shutdown_connections 2023-10-18T16:07:24.805 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.796+0000 7f35d6798700 1 --2- 172.21.15.151:0/653450009 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f35d0062650 0x7f35d014d450 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:24.806 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.796+0000 7f35d6798700 1 --2- 172.21.15.151:0/653450009 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f35d0063050 0x7f35d00b6870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:24.806 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.796+0000 7f35d6798700 1 -- 172.21.15.151:0/653450009 >> 172.21.15.151:0/653450009 conn(0x7f35d0064f00 msgr2=0x7f35d0065310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:24.806 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.796+0000 7f35d6798700 1 -- 172.21.15.151:0/653450009 shutdown_connections 2023-10-18T16:07:24.807 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.796+0000 7f35d6798700 1 -- 172.21.15.151:0/653450009 wait complete. 2023-10-18T16:07:24.809 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.798+0000 7f35d6798700 1 Processor -- start 2023-10-18T16:07:24.810 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.798+0000 7f35d6798700 1 -- start start 2023-10-18T16:07:24.810 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.799+0000 7f35d6798700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f35d0062650 0x7f35d015c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:24.810 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.799+0000 7f35d6798700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f35d0063050 0x7f35d015cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:24.811 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.800+0000 7f35d6798700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f35d00b6490 0x7f35d015f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:24.811 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.800+0000 7f35d6798700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35d015fed0 con 0x7f35d00b6490 2023-10-18T16:07:24.811 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.800+0000 7f35d6798700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35d015c190 con 0x7f35d0062650 2023-10-18T16:07:24.812 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.800+0000 7f35d6798700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35d00be830 con 0x7f35d0063050 2023-10-18T16:07:24.812 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.800+0000 7f35cf7fe700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f35d0063050 0x7f35d015cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:24.812 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.800+0000 7f35cf7fe700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f35d0063050 0x7f35d015cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:58434/0 (socket says 172.21.15.151:58434) 2023-10-18T16:07:24.813 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.800+0000 7f35cf7fe700 1 -- 172.21.15.151:0/3045241187 learned_addr learned my addr 172.21.15.151:0/3045241187 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-10-18T16:07:24.813 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.800+0000 7f35cffff700 1 --2- 172.21.15.151:0/3045241187 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f35d0062650 0x7f35d015c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:24.813 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.800+0000 7f35cf7fe700 1 -- 172.21.15.151:0/3045241187 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f35d0063050 msgr2=0x7f35d015cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-10-18T16:07:24.814 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.800+0000 7f35cf7fe700 1 -- 172.21.15.151:0/3045241187 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f35d0063050 msgr2=0x7f35d015cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-18T16:07:24.814 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.800+0000 7f35cf7fe700 1 --2- 172.21.15.151:0/3045241187 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f35d0063050 0x7f35d015cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-10-18T16:07:24.814 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.800+0000 7f35cf7fe700 1 --2- 172.21.15.151:0/3045241187 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f35d0063050 0x7f35d015cf00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:07:24.815 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.800+0000 7f35d4d35700 1 --2- 172.21.15.151:0/3045241187 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f35d00b6490 0x7f35d015f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:24.815 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.800+0000 7f35cffff700 1 -- 172.21.15.151:0/3045241187 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f35d0063050 msgr2=0x7f35d015cf00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-18T16:07:24.816 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.800+0000 7f35cffff700 1 --2- 172.21.15.151:0/3045241187 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f35d0063050 0x7f35d015cf00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:24.816 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.800+0000 7f35cffff700 1 -- 172.21.15.151:0/3045241187 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f35d00b6490 msgr2=0x7f35d015f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:24.816 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.800+0000 7f35cffff700 1 --2- 172.21.15.151:0/3045241187 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f35d00b6490 0x7f35d015f9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:24.817 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.800+0000 7f35cffff700 1 -- 172.21.15.151:0/3045241187 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35c8009a20 con 0x7f35d0062650 2023-10-18T16:07:24.817 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.800+0000 7f35cffff700 1 --2- 172.21.15.151:0/3045241187 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f35d0062650 0x7f35d015c620 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f35c4001eb0 tx=0x7f35c4007bb0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:24.817 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.800+0000 7f35b7fff700 1 -- 172.21.15.151:0/3045241187 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35c40072e0 con 0x7f35d0062650 2023-10-18T16:07:24.818 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.800+0000 7f35b7fff700 1 -- 172.21.15.151:0/3045241187 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35c4007440 con 0x7f35d0062650 2023-10-18T16:07:24.818 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.800+0000 7f35d6798700 1 -- 172.21.15.151:0/3045241187 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35d00bead0 con 0x7f35d0062650 2023-10-18T16:07:24.818 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.800+0000 7f35b7fff700 1 -- 172.21.15.151:0/3045241187 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35c40057a0 con 0x7f35d0062650 2023-10-18T16:07:24.819 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.800+0000 7f35d6798700 1 -- 172.21.15.151:0/3045241187 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35d015be20 con 0x7f35d0062650 2023-10-18T16:07:24.819 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.800+0000 7f35b7fff700 1 -- 172.21.15.151:0/3045241187 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 1) v1 ==== 495+0+0 (secure 0 0 0) 0x7f35c4013450 con 0x7f35d0062650 2023-10-18T16:07:24.820 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.800+0000 7f35b7fff700 1 -- 172.21.15.151:0/3045241187 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 984+0+0 (secure 0 0 0) 0x7f35c4005df0 con 0x7f35d0062650 2023-10-18T16:07:24.820 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.801+0000 7f35d6798700 1 -- 172.21.15.151:0/3045241187 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35d006c4e0 con 0x7f35d0062650 2023-10-18T16:07:24.820 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.803+0000 7f35b7fff700 1 -- 172.21.15.151:0/3045241187 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73265 (secure 0 0 0) 0x7f35c4018070 con 0x7f35d0062650 2023-10-18T16:07:24.886 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.882+0000 7f35d6798700 1 -- 172.21.15.151:0/3045241187 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd new", "uuid": "671895e2-57f3-4f1a-ba4a-a580af1f3ec2", "id": 2} v 0) v1 -- 0x7f35d00c07b0 con 0x7f35d0062650 2023-10-18T16:07:24.893 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.890+0000 7f35b7fff700 1 -- 172.21.15.151:0/3045241187 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "671895e2-57f3-4f1a-ba4a-a580af1f3ec2", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f35c4013930 con 0x7f35d0062650 2023-10-18T16:07:24.894 INFO:teuthology.orchestra.run.smithi151.stdout:2 2023-10-18T16:07:24.895 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.891+0000 7f35d6798700 1 -- 172.21.15.151:0/3045241187 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f35d0062650 msgr2=0x7f35d015c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:24.896 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.891+0000 7f35d6798700 1 --2- 172.21.15.151:0/3045241187 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f35d0062650 0x7f35d015c620 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f35c4001eb0 tx=0x7f35c4007bb0).stop 2023-10-18T16:07:24.896 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.891+0000 7f35d6798700 1 -- 172.21.15.151:0/3045241187 shutdown_connections 2023-10-18T16:07:24.896 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.891+0000 7f35d6798700 1 --2- 172.21.15.151:0/3045241187 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f35d00b6490 0x7f35d015f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:24.896 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.891+0000 7f35d6798700 1 --2- 172.21.15.151:0/3045241187 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f35d0062650 0x7f35d015c620 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:24.896 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.891+0000 7f35d6798700 1 --2- 172.21.15.151:0/3045241187 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f35d0063050 0x7f35d015cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:24.897 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.891+0000 7f35d6798700 1 -- 172.21.15.151:0/3045241187 >> 172.21.15.151:0/3045241187 conn(0x7f35d0064f00 msgr2=0x7f35d014eb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:24.897 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.891+0000 7f35d6798700 1 -- 172.21.15.151:0/3045241187 shutdown_connections 2023-10-18T16:07:24.897 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:24.891+0000 7f35d6798700 1 -- 172.21.15.151:0/3045241187 wait complete. 2023-10-18T16:07:24.910 DEBUG:teuthology.orchestra.run.smithi151:> sudo ceph --cluster ceph osd new 0c4e6e7f-7d9c-4e65-beec-fd0ca384f320 3 2023-10-18T16:07:25.024 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.020+0000 7fc8d3087700 1 Processor -- start 2023-10-18T16:07:25.024 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.020+0000 7fc8d3087700 1 -- start start 2023-10-18T16:07:25.027 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.021+0000 7fc8d3087700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc8cc14dad0 0x7fc8cc0c2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:25.027 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.021+0000 7fc8d3087700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fc8cc06a930 0x7fc8cc1542c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:25.027 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.021+0000 7fc8d3087700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc8cc0a8600 0x7fc8cc0a8a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:25.028 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.021+0000 7fc8d3087700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8cc14deb0 con 0x7fc8cc14dad0 2023-10-18T16:07:25.028 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.021+0000 7fc8d3087700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8cc150360 con 0x7fc8cc06a930 2023-10-18T16:07:25.028 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.021+0000 7fc8d3087700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8cc0a8f60 con 0x7fc8cc0a8600 2023-10-18T16:07:25.028 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.021+0000 7fc8d1624700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc8cc0a8600 0x7fc8cc0a8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:25.028 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.021+0000 7fc8cbfff700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fc8cc06a930 0x7fc8cc1542c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:25.029 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.021+0000 7fc8d1624700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc8cc0a8600 0x7fc8cc0a8a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:58440/0 (socket says 172.21.15.151:58440) 2023-10-18T16:07:25.029 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.021+0000 7fc8d1624700 1 -- 172.21.15.151:0/4195727342 learned_addr learned my addr 172.21.15.151:0/4195727342 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-10-18T16:07:25.029 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.021+0000 7fc8cbfff700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fc8cc06a930 0x7fc8cc1542c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:57274/0 (socket says 172.21.15.151:57274) 2023-10-18T16:07:25.029 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.021+0000 7fc8d1624700 1 -- 172.21.15.151:0/4195727342 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc8cc0a8600 msgr2=0x7fc8cc0a8a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-18T16:07:25.030 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.021+0000 7fc8d1624700 1 -- 172.21.15.151:0/4195727342 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc8cc0a8600 msgr2=0x7fc8cc0a8a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-18T16:07:25.030 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.021+0000 7fc8d1624700 1 --2- 172.21.15.151:0/4195727342 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc8cc0a8600 0x7fc8cc0a8a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-10-18T16:07:25.030 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.021+0000 7fc8d0e23700 1 --2- 172.21.15.151:0/4195727342 >> v2:172.21.15.88:3300/0 conn(0x7fc8cc14dad0 0x7fc8cc0c2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:25.030 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.021+0000 7fc8d1624700 1 --2- 172.21.15.151:0/4195727342 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc8cc0a8600 0x7fc8cc0a8a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:07:25.030 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.021+0000 7fc8cbfff700 1 -- 172.21.15.151:0/4195727342 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc8cc0a8600 msgr2=0x7fc8cc0a8a70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:07:25.031 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.021+0000 7fc8cbfff700 1 --2- 172.21.15.151:0/4195727342 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc8cc0a8600 0x7fc8cc0a8a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:25.031 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.021+0000 7fc8cbfff700 1 -- 172.21.15.151:0/4195727342 >> v2:172.21.15.88:3300/0 conn(0x7fc8cc14dad0 msgr2=0x7fc8cc0c2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:25.031 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.021+0000 7fc8cbfff700 1 --2- 172.21.15.151:0/4195727342 >> v2:172.21.15.88:3300/0 conn(0x7fc8cc14dad0 0x7fc8cc0c2ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:25.031 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.021+0000 7fc8cbfff700 1 -- 172.21.15.151:0/4195727342 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8cc150550 con 0x7fc8cc06a930 2023-10-18T16:07:25.031 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.022+0000 7fc8cbfff700 1 --2- 172.21.15.151:0/4195727342 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc8cc06a930 0x7fc8cc1542c0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fc8cc0c2c50 tx=0x7fc8b8009060).ready entity=mon.1 client_cookie=a947ee21bda8b503 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:25.032 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.022+0000 7fc8cb7fe700 1 -- 172.21.15.151:0/4195727342 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8b800f510 con 0x7fc8cc06a930 2023-10-18T16:07:25.032 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.022+0000 7fc8cb7fe700 1 -- 172.21.15.151:0/4195727342 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8b800f670 con 0x7fc8cc06a930 2023-10-18T16:07:25.032 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.022+0000 7fc8cb7fe700 1 -- 172.21.15.151:0/4195727342 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8b800f9f0 con 0x7fc8cc06a930 2023-10-18T16:07:25.032 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.022+0000 7fc8d3087700 1 -- 172.21.15.151:0/4195727342 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc8cc06a930 msgr2=0x7fc8cc1542c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:25.032 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.022+0000 7fc8d3087700 1 --2- 172.21.15.151:0/4195727342 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc8cc06a930 0x7fc8cc1542c0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fc8cc0c2c50 tx=0x7fc8b8009060).stop 2023-10-18T16:07:25.033 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.022+0000 7fc8d3087700 1 -- 172.21.15.151:0/4195727342 shutdown_connections 2023-10-18T16:07:25.033 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.022+0000 7fc8d3087700 1 --2- 172.21.15.151:0/4195727342 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc8cc0a8600 0x7fc8cc0a8a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:25.033 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.022+0000 7fc8d3087700 1 --2- 172.21.15.151:0/4195727342 >> v2:172.21.15.88:3300/0 conn(0x7fc8cc14dad0 0x7fc8cc0c2ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:25.033 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.022+0000 7fc8d3087700 1 --2- 172.21.15.151:0/4195727342 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc8cc06a930 0x7fc8cc1542c0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:25.033 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.022+0000 7fc8d3087700 1 -- 172.21.15.151:0/4195727342 >> 172.21.15.151:0/4195727342 conn(0x7fc8cc069d80 msgr2=0x7fc8cc06a190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:25.033 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.022+0000 7fc8d3087700 1 -- 172.21.15.151:0/4195727342 shutdown_connections 2023-10-18T16:07:25.034 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.022+0000 7fc8d3087700 1 -- 172.21.15.151:0/4195727342 wait complete. 2023-10-18T16:07:25.034 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.023+0000 7fc8d3087700 1 Processor -- start 2023-10-18T16:07:25.034 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.024+0000 7fc8d3087700 1 -- start start 2023-10-18T16:07:25.037 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.024+0000 7fc8d3087700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc8cc06a930 0x7fc8cc0c4d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:25.037 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.024+0000 7fc8d3087700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8cc0a8600 0x7fc8cc1462e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:25.037 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.024+0000 7fc8d3087700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc8cc14dad0 0x7fc8cc0bc310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:25.037 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.024+0000 7fc8d3087700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8cc0c77b0 con 0x7fc8cc0a8600 2023-10-18T16:07:25.037 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.024+0000 7fc8d3087700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8cc148990 con 0x7fc8cc06a930 2023-10-18T16:07:25.038 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.024+0000 7fc8d3087700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8cc0c54d0 con 0x7fc8cc14dad0 2023-10-18T16:07:25.038 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.024+0000 7fc8d1624700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc8cc14dad0 0x7fc8cc0bc310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:25.038 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.024+0000 7fc8d0e23700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc8cc06a930 0x7fc8cc0c4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:25.038 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.024+0000 7fc8d0e23700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc8cc06a930 0x7fc8cc0c4d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:57278/0 (socket says 172.21.15.151:57278) 2023-10-18T16:07:25.038 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.024+0000 7fc8d1624700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc8cc14dad0 0x7fc8cc0bc310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:58446/0 (socket says 172.21.15.151:58446) 2023-10-18T16:07:25.039 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.024+0000 7fc8d0e23700 1 -- 172.21.15.151:0/1680968222 learned_addr learned my addr 172.21.15.151:0/1680968222 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-10-18T16:07:25.039 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.024+0000 7fc8cbfff700 1 --2- 172.21.15.151:0/1680968222 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8cc0a8600 0x7fc8cc1462e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:25.039 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.024+0000 7fc8d1624700 1 -- 172.21.15.151:0/1680968222 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc8cc14dad0 msgr2=0x7fc8cc0bc310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-10-18T16:07:25.039 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.024+0000 7fc8d1624700 1 -- 172.21.15.151:0/1680968222 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc8cc14dad0 msgr2=0x7fc8cc0bc310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-18T16:07:25.039 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.024+0000 7fc8d1624700 1 --2- 172.21.15.151:0/1680968222 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc8cc14dad0 0x7fc8cc0bc310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-10-18T16:07:25.040 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.025+0000 7fc8d1624700 1 --2- 172.21.15.151:0/1680968222 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc8cc14dad0 0x7fc8cc0bc310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:07:25.040 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.025+0000 7fc8d0e23700 1 -- 172.21.15.151:0/1680968222 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc8cc14dad0 msgr2=0x7fc8cc0bc310 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-18T16:07:25.040 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.025+0000 7fc8d0e23700 1 --2- 172.21.15.151:0/1680968222 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc8cc14dad0 0x7fc8cc0bc310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:25.040 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.025+0000 7fc8d0e23700 1 -- 172.21.15.151:0/1680968222 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8cc0a8600 msgr2=0x7fc8cc1462e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:25.040 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.025+0000 7fc8d0e23700 1 --2- 172.21.15.151:0/1680968222 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8cc0a8600 0x7fc8cc1462e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:25.041 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.025+0000 7fc8d0e23700 1 -- 172.21.15.151:0/1680968222 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8b80092a0 con 0x7fc8cc06a930 2023-10-18T16:07:25.041 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.025+0000 7fc8d0e23700 1 --2- 172.21.15.151:0/1680968222 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc8cc06a930 0x7fc8cc0c4d10 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fc8c000e930 tx=0x7fc8c0009060).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:25.041 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.025+0000 7fc8c8ff9700 1 -- 172.21.15.151:0/1680968222 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c0009510 con 0x7fc8cc06a930 2023-10-18T16:07:25.041 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.025+0000 7fc8c8ff9700 1 -- 172.21.15.151:0/1680968222 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8c000c670 con 0x7fc8cc06a930 2023-10-18T16:07:25.041 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.025+0000 7fc8d3087700 1 -- 172.21.15.151:0/1680968222 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8cc148c50 con 0x7fc8cc06a930 2023-10-18T16:07:25.042 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.025+0000 7fc8c8ff9700 1 -- 172.21.15.151:0/1680968222 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c000c9f0 con 0x7fc8cc06a930 2023-10-18T16:07:25.042 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.025+0000 7fc8d3087700 1 -- 172.21.15.151:0/1680968222 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8cc0c74a0 con 0x7fc8cc06a930 2023-10-18T16:07:25.042 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.025+0000 7fc8c8ff9700 1 -- 172.21.15.151:0/1680968222 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 1) v1 ==== 495+0+0 (secure 0 0 0) 0x7fc8c001b400 con 0x7fc8cc06a930 2023-10-18T16:07:25.042 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.025+0000 7fc8c8ff9700 1 -- 172.21.15.151:0/1680968222 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1099+0+0 (secure 0 0 0) 0x7fc8c00128b0 con 0x7fc8cc06a930 2023-10-18T16:07:25.043 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.026+0000 7fc8d3087700 1 -- 172.21.15.151:0/1680968222 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8cc0bbc00 con 0x7fc8cc06a930 2023-10-18T16:07:25.043 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.028+0000 7fc8c8ff9700 1 -- 172.21.15.151:0/1680968222 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73265 (secure 0 0 0) 0x7fc8c003ae50 con 0x7fc8cc06a930 2023-10-18T16:07:25.123 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.120+0000 7fc8d3087700 1 -- 172.21.15.151:0/1680968222 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd new", "uuid": "0c4e6e7f-7d9c-4e65-beec-fd0ca384f320", "id": 3} v 0) v1 -- 0x7fc8cc0a92b0 con 0x7fc8cc06a930 2023-10-18T16:07:25.131 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.128+0000 7fc8c8ff9700 1 -- 172.21.15.151:0/1680968222 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0c4e6e7f-7d9c-4e65-beec-fd0ca384f320", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7fc8c0029030 con 0x7fc8cc06a930 2023-10-18T16:07:25.132 INFO:teuthology.orchestra.run.smithi151.stdout:3 2023-10-18T16:07:25.133 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.129+0000 7fc8d3087700 1 -- 172.21.15.151:0/1680968222 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc8cc06a930 msgr2=0x7fc8cc0c4d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:25.133 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.129+0000 7fc8d3087700 1 --2- 172.21.15.151:0/1680968222 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc8cc06a930 0x7fc8cc0c4d10 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fc8c000e930 tx=0x7fc8c0009060).stop 2023-10-18T16:07:25.133 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.129+0000 7fc8d3087700 1 -- 172.21.15.151:0/1680968222 shutdown_connections 2023-10-18T16:07:25.134 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.129+0000 7fc8d3087700 1 --2- 172.21.15.151:0/1680968222 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc8cc14dad0 0x7fc8cc0bc310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:25.134 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.129+0000 7fc8d3087700 1 --2- 172.21.15.151:0/1680968222 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc8cc06a930 0x7fc8cc0c4d10 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:25.134 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.129+0000 7fc8d3087700 1 --2- 172.21.15.151:0/1680968222 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8cc0a8600 0x7fc8cc1462e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:25.134 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.129+0000 7fc8d3087700 1 -- 172.21.15.151:0/1680968222 >> 172.21.15.151:0/1680968222 conn(0x7fc8cc069d80 msgr2=0x7fc8cc0c2480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:25.134 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.129+0000 7fc8d3087700 1 -- 172.21.15.151:0/1680968222 shutdown_connections 2023-10-18T16:07:25.135 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.129+0000 7fc8d3087700 1 -- 172.21.15.151:0/1680968222 wait complete. 2023-10-18T16:07:25.141 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-10-18T16:07:25.138+0000 7f07f6041680 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-18T16:07:25.147 DEBUG:teuthology.orchestra.run.smithi151:> sudo ceph --cluster ceph osd new 29570691-ef35-4b80-9247-aa3703877af4 4 2023-10-18T16:07:25.191 INFO:tasks.ceph.mgr.x.smithi151.stderr:2023-10-18T16:07:25.188+0000 7f8d9db54680 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-18T16:07:25.209 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-10-18T16:07:25.206+0000 7f07f6041680 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-18T16:07:25.260 INFO:tasks.ceph.mgr.x.smithi151.stderr:2023-10-18T16:07:25.256+0000 7f8d9db54680 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-18T16:07:25.264 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.261+0000 7fe5df59e700 1 Processor -- start 2023-10-18T16:07:25.265 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.261+0000 7fe5df59e700 1 -- start start 2023-10-18T16:07:25.267 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.262+0000 7fe5df59e700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fe5e0149e80 0x7fe5e014a260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:25.267 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.262+0000 7fe5df59e700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5e00bb980 con 0x7fe5e00c66d0 2023-10-18T16:07:25.267 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.262+0000 7fe5df59e700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5e00bbc60 con 0x7fe5e0150b70 2023-10-18T16:07:25.268 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.262+0000 7fe5df59e700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5e0064b60 con 0x7fe5e0149e80 2023-10-18T16:07:25.268 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.262+0000 7fe5de59c700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fe5e0149e80 0x7fe5e014a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:25.268 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.262+0000 7fe5ddd9b700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fe5e0150b70 0x7fe5e00c1160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:58232/0 (socket says 172.21.15.151:58232) 2023-10-18T16:07:25.268 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.262+0000 7fe5de59c700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fe5e0149e80 0x7fe5e014a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:57284/0 (socket says 172.21.15.151:57284) 2023-10-18T16:07:25.268 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.262+0000 7fe5de59c700 1 -- 172.21.15.151:0/2730666733 learned_addr learned my addr 172.21.15.151:0/2730666733 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-10-18T16:07:25.269 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.262+0000 7fe5de59c700 1 -- 172.21.15.151:0/2730666733 >> v1:172.21.15.151:6789/0 conn(0x7fe5e0150b70 legacy=0x7fe5e00c1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:25.269 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.262+0000 7fe5de59c700 1 -- 172.21.15.151:0/2730666733 >> v1:172.21.15.88:6789/0 conn(0x7fe5e00c66d0 legacy=0x7fe5e00c2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:25.269 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.262+0000 7fe5dd59a700 1 -- 172.21.15.151:0/2730666733 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 655025540 0 0) 0x7fe5e00bbc60 con 0x7fe5e0150b70 2023-10-18T16:07:25.269 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.262+0000 7fe5de59c700 1 -- 172.21.15.151:0/2730666733 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5e00bbf40 con 0x7fe5e0149e80 2023-10-18T16:07:25.269 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.262+0000 7fe5de59c700 1 --2- 172.21.15.151:0/2730666733 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe5e0149e80 0x7fe5e014a260 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7fe5d0006fd0 tx=0x7fe5d00090f0).ready entity=mon.1 client_cookie=b360645d2d3b5f11 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:25.270 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.262+0000 7fe5dd59a700 1 -- 172.21.15.151:0/2730666733 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d000b600 con 0x7fe5e0149e80 2023-10-18T16:07:25.270 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.262+0000 7fe5dd59a700 1 -- 172.21.15.151:0/2730666733 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5d000b760 con 0x7fe5e0149e80 2023-10-18T16:07:25.270 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.262+0000 7fe5dd59a700 1 -- 172.21.15.151:0/2730666733 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d000bae0 con 0x7fe5e0149e80 2023-10-18T16:07:25.270 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.263+0000 7fe5df59e700 1 -- 172.21.15.151:0/2730666733 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe5e0149e80 msgr2=0x7fe5e014a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:25.270 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.263+0000 7fe5df59e700 1 --2- 172.21.15.151:0/2730666733 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe5e0149e80 0x7fe5e014a260 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7fe5d0006fd0 tx=0x7fe5d00090f0).stop 2023-10-18T16:07:25.270 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.263+0000 7fe5df59e700 1 -- 172.21.15.151:0/2730666733 shutdown_connections 2023-10-18T16:07:25.271 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.263+0000 7fe5df59e700 1 --2- 172.21.15.151:0/2730666733 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe5e0149e80 0x7fe5e014a260 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:25.271 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.263+0000 7fe5df59e700 1 -- 172.21.15.151:0/2730666733 >> 172.21.15.151:0/2730666733 conn(0x7fe5e005e050 msgr2=0x7fe5e005e460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:25.271 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.263+0000 7fe5df59e700 1 -- 172.21.15.151:0/2730666733 shutdown_connections 2023-10-18T16:07:25.271 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.263+0000 7fe5df59e700 1 -- 172.21.15.151:0/2730666733 wait complete. 2023-10-18T16:07:25.274 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.264+0000 7fe5df59e700 1 Processor -- start 2023-10-18T16:07:25.274 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.264+0000 7fe5df59e700 1 -- start start 2023-10-18T16:07:25.274 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.265+0000 7fe5df59e700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5e00c66d0 0x7fe5e0154b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:25.275 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.265+0000 7fe5df59e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe5e0149e80 0x7fe5e01564e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:25.275 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.265+0000 7fe5df59e700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe5e0150b70 0x7fe5e0160bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:25.275 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.265+0000 7fe5df59e700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e0156130 con 0x7fe5e00c66d0 2023-10-18T16:07:25.275 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.265+0000 7fe5df59e700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e0154670 con 0x7fe5e0149e80 2023-10-18T16:07:25.275 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.265+0000 7fe5df59e700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e01610b0 con 0x7fe5e0150b70 2023-10-18T16:07:25.275 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.265+0000 7fe5ddd9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe5e0149e80 0x7fe5e01564e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:25.276 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.265+0000 7fe5ded9d700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe5e0150b70 0x7fe5e0160bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:25.276 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.265+0000 7fe5ddd9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe5e0149e80 0x7fe5e01564e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:57292/0 (socket says 172.21.15.151:57292) 2023-10-18T16:07:25.276 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.265+0000 7fe5ded9d700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe5e0150b70 0x7fe5e0160bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:58460/0 (socket says 172.21.15.151:58460) 2023-10-18T16:07:25.276 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.265+0000 7fe5ddd9b700 1 -- 172.21.15.151:0/2184526155 learned_addr learned my addr 172.21.15.151:0/2184526155 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-10-18T16:07:25.276 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.265+0000 7fe5de59c700 1 --2- 172.21.15.151:0/2184526155 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5e00c66d0 0x7fe5e0154b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:25.277 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.265+0000 7fe5ded9d700 1 -- 172.21.15.151:0/2184526155 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe5e0150b70 msgr2=0x7fe5e0160bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2023-10-18T16:07:25.277 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.265+0000 7fe5ded9d700 1 -- 172.21.15.151:0/2184526155 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe5e0150b70 msgr2=0x7fe5e0160bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-18T16:07:25.277 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.265+0000 7fe5ded9d700 1 --2- 172.21.15.151:0/2184526155 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe5e0150b70 0x7fe5e0160bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-10-18T16:07:25.277 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.265+0000 7fe5ded9d700 1 --2- 172.21.15.151:0/2184526155 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe5e0150b70 0x7fe5e0160bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:07:25.277 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.265+0000 7fe5ddd9b700 1 -- 172.21.15.151:0/2184526155 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe5e0150b70 msgr2=0x7fe5e0160bc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-18T16:07:25.278 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.265+0000 7fe5ddd9b700 1 --2- 172.21.15.151:0/2184526155 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe5e0150b70 0x7fe5e0160bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:25.278 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.265+0000 7fe5ddd9b700 1 -- 172.21.15.151:0/2184526155 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5e00c66d0 msgr2=0x7fe5e0154b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:25.278 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.265+0000 7fe5ddd9b700 1 --2- 172.21.15.151:0/2184526155 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5e00c66d0 0x7fe5e0154b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:25.278 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.265+0000 7fe5ddd9b700 1 -- 172.21.15.151:0/2184526155 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5d0009370 con 0x7fe5e0149e80 2023-10-18T16:07:25.278 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.265+0000 7fe5de59c700 1 --2- 172.21.15.151:0/2184526155 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5e00c66d0 0x7fe5e0154b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:07:25.279 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.265+0000 7fe5ddd9b700 1 --2- 172.21.15.151:0/2184526155 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe5e0149e80 0x7fe5e01564e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7fe5d400c4c0 tx=0x7fe5d4007c10).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:25.279 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.266+0000 7fe5ce7fc700 1 -- 172.21.15.151:0/2184526155 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d40072e0 con 0x7fe5e0149e80 2023-10-18T16:07:25.279 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.266+0000 7fe5ce7fc700 1 -- 172.21.15.151:0/2184526155 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5d4007440 con 0x7fe5e0149e80 2023-10-18T16:07:25.279 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.266+0000 7fe5ce7fc700 1 -- 172.21.15.151:0/2184526155 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d4005930 con 0x7fe5e0149e80 2023-10-18T16:07:25.279 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.266+0000 7fe5df59e700 1 -- 172.21.15.151:0/2184526155 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5e00bf820 con 0x7fe5e0149e80 2023-10-18T16:07:25.280 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.266+0000 7fe5df59e700 1 -- 172.21.15.151:0/2184526155 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5e014a4d0 con 0x7fe5e0149e80 2023-10-18T16:07:25.280 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.266+0000 7fe5ce7fc700 1 -- 172.21.15.151:0/2184526155 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 1) v1 ==== 495+0+0 (secure 0 0 0) 0x7fe5d4013450 con 0x7fe5e0149e80 2023-10-18T16:07:25.280 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.266+0000 7fe5ce7fc700 1 -- 172.21.15.151:0/2184526155 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1214+0+0 (secure 0 0 0) 0x7fe5d401d920 con 0x7fe5e0149e80 2023-10-18T16:07:25.280 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.267+0000 7fe5df59e700 1 -- 172.21.15.151:0/2184526155 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5e0154820 con 0x7fe5e0149e80 2023-10-18T16:07:25.280 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.269+0000 7fe5ce7fc700 1 -- 172.21.15.151:0/2184526155 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73265 (secure 0 0 0) 0x7fe5e0154820 con 0x7fe5e0149e80 2023-10-18T16:07:25.351 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.347+0000 7fe5df59e700 1 -- 172.21.15.151:0/2184526155 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd new", "uuid": "29570691-ef35-4b80-9247-aa3703877af4", "id": 4} v 0) v1 -- 0x7fe5e0084180 con 0x7fe5e0149e80 2023-10-18T16:07:25.358 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.355+0000 7fe5ce7fc700 1 -- 172.21.15.151:0/2184526155 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "29570691-ef35-4b80-9247-aa3703877af4", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7fe5d401dbf0 con 0x7fe5e0149e80 2023-10-18T16:07:25.359 INFO:teuthology.orchestra.run.smithi151.stdout:4 2023-10-18T16:07:25.360 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.356+0000 7fe5df59e700 1 -- 172.21.15.151:0/2184526155 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe5e0149e80 msgr2=0x7fe5e01564e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:25.360 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.356+0000 7fe5df59e700 1 --2- 172.21.15.151:0/2184526155 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe5e0149e80 0x7fe5e01564e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7fe5d400c4c0 tx=0x7fe5d4007c10).stop 2023-10-18T16:07:25.360 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.356+0000 7fe5df59e700 1 -- 172.21.15.151:0/2184526155 shutdown_connections 2023-10-18T16:07:25.360 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.356+0000 7fe5df59e700 1 --2- 172.21.15.151:0/2184526155 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe5e0150b70 0x7fe5e0160bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:25.361 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.356+0000 7fe5df59e700 1 --2- 172.21.15.151:0/2184526155 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5e00c66d0 0x7fe5e0154b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:25.361 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.356+0000 7fe5df59e700 1 --2- 172.21.15.151:0/2184526155 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe5e0149e80 0x7fe5e01564e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:25.361 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.356+0000 7fe5df59e700 1 -- 172.21.15.151:0/2184526155 >> 172.21.15.151:0/2184526155 conn(0x7fe5e005e050 msgr2=0x7fe5e00c45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:25.361 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.356+0000 7fe5df59e700 1 -- 172.21.15.151:0/2184526155 shutdown_connections 2023-10-18T16:07:25.362 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.356+0000 7fe5df59e700 1 -- 172.21.15.151:0/2184526155 wait complete. 2023-10-18T16:07:25.371 DEBUG:teuthology.orchestra.run.smithi151:> sudo ceph --cluster ceph osd new 26540b7b-032a-4b3b-bf6c-b131a449fa99 5 2023-10-18T16:07:25.403 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-10-18T16:07:25.400+0000 7f07f6041680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-18T16:07:25.455 INFO:tasks.ceph.mgr.x.smithi151.stderr:2023-10-18T16:07:25.451+0000 7f8d9db54680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-18T16:07:25.524 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.520+0000 7f988c745700 1 Processor -- start 2023-10-18T16:07:25.525 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.520+0000 7f988c745700 1 -- start start 2023-10-18T16:07:25.530 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.521+0000 7f988c745700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f9884069d40 0x7f988406a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:25.530 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.521+0000 7f988c745700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9884068050 0x7f98840c01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:25.530 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.521+0000 7f988c745700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f98840c0690 0x7f9884150780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:25.531 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.521+0000 7f988c745700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9884150280 con 0x7f9884068050 2023-10-18T16:07:25.531 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.521+0000 7f988c745700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98841539f0 con 0x7f9884069d40 2023-10-18T16:07:25.531 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.521+0000 7f988c745700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9884153cd0 con 0x7f98840c0690 2023-10-18T16:07:25.531 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.522+0000 7f988ace2700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f98840c0690 0x7f9884150780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:25.532 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.522+0000 7f988a4e1700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f9884069d40 0x7f988406a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:25.532 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.522+0000 7f988ace2700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f98840c0690 0x7f9884150780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:58466/0 (socket says 172.21.15.151:58466) 2023-10-18T16:07:25.532 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.522+0000 7f988ace2700 1 -- 172.21.15.151:0/2701886412 learned_addr learned my addr 172.21.15.151:0/2701886412 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-10-18T16:07:25.532 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.522+0000 7f988a4e1700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f9884069d40 0x7f988406a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:57296/0 (socket says 172.21.15.151:57296) 2023-10-18T16:07:25.532 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.522+0000 7f9889ce0700 1 --2- 172.21.15.151:0/2701886412 >> v2:172.21.15.88:3300/0 conn(0x7f9884068050 0x7f98840c01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:25.533 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.522+0000 7f988ace2700 1 -- 172.21.15.151:0/2701886412 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f98840c0690 msgr2=0x7f9884150780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-18T16:07:25.533 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.522+0000 7f988ace2700 1 -- 172.21.15.151:0/2701886412 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f98840c0690 msgr2=0x7f9884150780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-18T16:07:25.533 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.522+0000 7f988ace2700 1 --2- 172.21.15.151:0/2701886412 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f98840c0690 0x7f9884150780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-10-18T16:07:25.533 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.522+0000 7f988ace2700 1 --2- 172.21.15.151:0/2701886412 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f98840c0690 0x7f9884150780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:07:25.533 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.522+0000 7f988a4e1700 1 -- 172.21.15.151:0/2701886412 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f98840c0690 msgr2=0x7f9884150780 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:07:25.534 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.522+0000 7f988a4e1700 1 --2- 172.21.15.151:0/2701886412 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f98840c0690 0x7f9884150780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:25.534 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.522+0000 7f988a4e1700 1 -- 172.21.15.151:0/2701886412 >> v2:172.21.15.88:3300/0 conn(0x7f9884068050 msgr2=0x7f98840c01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:25.534 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.522+0000 7f988a4e1700 1 --2- 172.21.15.151:0/2701886412 >> v2:172.21.15.88:3300/0 conn(0x7f9884068050 0x7f98840c01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:25.534 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.522+0000 7f988a4e1700 1 -- 172.21.15.151:0/2701886412 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9884153fb0 con 0x7f9884069d40 2023-10-18T16:07:25.534 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.522+0000 7f988a4e1700 1 --2- 172.21.15.151:0/2701886412 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9884069d40 0x7f988406a120 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f9884065a50 tx=0x7f9874009060).ready entity=mon.1 client_cookie=723df1753878a9e5 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:25.534 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.522+0000 7f98894df700 1 -- 172.21.15.151:0/2701886412 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f987400f510 con 0x7f9884069d40 2023-10-18T16:07:25.535 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.522+0000 7f98894df700 1 -- 172.21.15.151:0/2701886412 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f987400f670 con 0x7f9884069d40 2023-10-18T16:07:25.535 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.522+0000 7f98894df700 1 -- 172.21.15.151:0/2701886412 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f987400f9f0 con 0x7f9884069d40 2023-10-18T16:07:25.535 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.523+0000 7f988c745700 1 -- 172.21.15.151:0/2701886412 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9884069d40 msgr2=0x7f988406a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:25.535 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.523+0000 7f988c745700 1 --2- 172.21.15.151:0/2701886412 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9884069d40 0x7f988406a120 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f9884065a50 tx=0x7f9874009060).stop 2023-10-18T16:07:25.536 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.523+0000 7f988c745700 1 -- 172.21.15.151:0/2701886412 shutdown_connections 2023-10-18T16:07:25.536 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.523+0000 7f988c745700 1 --2- 172.21.15.151:0/2701886412 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9884069d40 0x7f988406a120 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:25.536 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.523+0000 7f988c745700 1 --2- 172.21.15.151:0/2701886412 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f98840c0690 0x7f9884150780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:25.536 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.523+0000 7f988c745700 1 --2- 172.21.15.151:0/2701886412 >> v2:172.21.15.88:3300/0 conn(0x7f9884068050 0x7f98840c01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:25.536 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.523+0000 7f988c745700 1 -- 172.21.15.151:0/2701886412 >> 172.21.15.151:0/2701886412 conn(0x7f988405e050 msgr2=0x7f9884064f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:25.537 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.523+0000 7f988c745700 1 -- 172.21.15.151:0/2701886412 shutdown_connections 2023-10-18T16:07:25.537 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.523+0000 7f988c745700 1 -- 172.21.15.151:0/2701886412 wait complete. 2023-10-18T16:07:25.537 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.524+0000 7f988c745700 1 Processor -- start 2023-10-18T16:07:25.537 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.524+0000 7f988c745700 1 -- start start 2023-10-18T16:07:25.537 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.525+0000 7f988c745700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9884068050 0x7f9884160670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:25.537 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.525+0000 7f988c745700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9884069d40 0x7f988414e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:25.538 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.525+0000 7f988c745700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f98840c0690 0x7f98840bdff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:25.538 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.525+0000 7f988c745700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98840bbf50 con 0x7f9884068050 2023-10-18T16:07:25.538 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.525+0000 7f988c745700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f988414ddf0 con 0x7f98840c0690 2023-10-18T16:07:25.538 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.525+0000 7f988c745700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9884160b60 con 0x7f9884069d40 2023-10-18T16:07:25.538 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.525+0000 7f988ace2700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f98840c0690 0x7f98840bdff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:25.539 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.525+0000 7f9889ce0700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9884069d40 0x7f988414e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:25.539 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.525+0000 7f988ace2700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f98840c0690 0x7f98840bdff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:57304/0 (socket says 172.21.15.151:57304) 2023-10-18T16:07:25.539 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.525+0000 7f988ace2700 1 -- 172.21.15.151:0/633867982 learned_addr learned my addr 172.21.15.151:0/633867982 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-10-18T16:07:25.539 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.525+0000 7f988a4e1700 1 --2- 172.21.15.151:0/633867982 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9884068050 0x7f9884160670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:25.539 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.525+0000 7f9889ce0700 1 -- 172.21.15.151:0/633867982 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9884069d40 msgr2=0x7f988414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-10-18T16:07:25.540 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.525+0000 7f9889ce0700 1 -- 172.21.15.151:0/633867982 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9884069d40 msgr2=0x7f988414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-18T16:07:25.540 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.525+0000 7f9889ce0700 1 --2- 172.21.15.151:0/633867982 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9884069d40 0x7f988414e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-10-18T16:07:25.540 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.525+0000 7f9889ce0700 1 --2- 172.21.15.151:0/633867982 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9884069d40 0x7f988414e280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:07:25.540 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.525+0000 7f988ace2700 1 -- 172.21.15.151:0/633867982 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9884069d40 msgr2=0x7f988414e280 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-18T16:07:25.540 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.525+0000 7f988ace2700 1 --2- 172.21.15.151:0/633867982 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9884069d40 0x7f988414e280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:25.540 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.525+0000 7f988ace2700 1 -- 172.21.15.151:0/633867982 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9884068050 msgr2=0x7f9884160670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:25.541 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.525+0000 7f988ace2700 1 --2- 172.21.15.151:0/633867982 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9884068050 0x7f9884160670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:25.541 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.525+0000 7f988ace2700 1 -- 172.21.15.151:0/633867982 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98740092a0 con 0x7f98840c0690 2023-10-18T16:07:25.541 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.525+0000 7f988ace2700 1 --2- 172.21.15.151:0/633867982 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f98840c0690 0x7f98840bdff0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f987c007bd0 tx=0x7f987c0092d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:25.541 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.525+0000 7f987a7fc700 1 -- 172.21.15.151:0/633867982 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f987c00d8a0 con 0x7f98840c0690 2023-10-18T16:07:25.541 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.525+0000 7f987a7fc700 1 -- 172.21.15.151:0/633867982 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f987c00f040 con 0x7f98840c0690 2023-10-18T16:07:25.541 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.525+0000 7f988c745700 1 -- 172.21.15.151:0/633867982 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f988414e770 con 0x7f98840c0690 2023-10-18T16:07:25.542 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.525+0000 7f987a7fc700 1 -- 172.21.15.151:0/633867982 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f987c00dc50 con 0x7f98840c0690 2023-10-18T16:07:25.542 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.525+0000 7f988c745700 1 -- 172.21.15.151:0/633867982 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f988414d4c0 con 0x7f98840c0690 2023-10-18T16:07:25.542 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.525+0000 7f987a7fc700 1 -- 172.21.15.151:0/633867982 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 1) v1 ==== 495+0+0 (secure 0 0 0) 0x7f987c003020 con 0x7f98840c0690 2023-10-18T16:07:25.542 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.525+0000 7f987a7fc700 1 -- 172.21.15.151:0/633867982 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1329+0+0 (secure 0 0 0) 0x7f987c0139a0 con 0x7f98840c0690 2023-10-18T16:07:25.542 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.526+0000 7f988c745700 1 -- 172.21.15.151:0/633867982 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f988406c4e0 con 0x7f98840c0690 2023-10-18T16:07:25.543 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.528+0000 7f987a7fc700 1 -- 172.21.15.151:0/633867982 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73265 (secure 0 0 0) 0x7f987c021060 con 0x7f98840c0690 2023-10-18T16:07:25.612 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.608+0000 7f988c745700 1 -- 172.21.15.151:0/633867982 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd new", "uuid": "26540b7b-032a-4b3b-bf6c-b131a449fa99", "id": 5} v 0) v1 -- 0x7f988414df30 con 0x7f98840c0690 2023-10-18T16:07:25.620 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.616+0000 7f987a7fc700 1 -- 172.21.15.151:0/633867982 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "26540b7b-032a-4b3b-bf6c-b131a449fa99", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f987c013c30 con 0x7f98840c0690 2023-10-18T16:07:25.620 INFO:teuthology.orchestra.run.smithi151.stdout:5 2023-10-18T16:07:25.621 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.617+0000 7f988c745700 1 -- 172.21.15.151:0/633867982 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f98840c0690 msgr2=0x7f98840bdff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:25.622 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.617+0000 7f988c745700 1 --2- 172.21.15.151:0/633867982 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f98840c0690 0x7f98840bdff0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f987c007bd0 tx=0x7f987c0092d0).stop 2023-10-18T16:07:25.622 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.617+0000 7f988c745700 1 -- 172.21.15.151:0/633867982 shutdown_connections 2023-10-18T16:07:25.622 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.617+0000 7f988c745700 1 --2- 172.21.15.151:0/633867982 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f98840c0690 0x7f98840bdff0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:25.622 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.617+0000 7f988c745700 1 --2- 172.21.15.151:0/633867982 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9884068050 0x7f9884160670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:25.623 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.617+0000 7f988c745700 1 --2- 172.21.15.151:0/633867982 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9884069d40 0x7f988414e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:25.623 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.617+0000 7f988c745700 1 -- 172.21.15.151:0/633867982 >> 172.21.15.151:0/633867982 conn(0x7f988405e050 msgr2=0x7f9884064f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:25.623 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.617+0000 7f988c745700 1 -- 172.21.15.151:0/633867982 shutdown_connections 2023-10-18T16:07:25.623 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.617+0000 7f988c745700 1 -- 172.21.15.151:0/633867982 wait complete. 2023-10-18T16:07:25.632 DEBUG:teuthology.orchestra.run.smithi151:> sudo ceph --cluster ceph osd new 7af01f9c-90db-45fd-93c8-09517c9b0881 6 2023-10-18T16:07:25.748 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.744+0000 7f8f0debe700 1 Processor -- start 2023-10-18T16:07:25.748 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.744+0000 7f8f0debe700 1 -- start start 2023-10-18T16:07:25.753 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.745+0000 7f8f0debe700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8f080bf930 0x7f8f08066cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:25.753 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.745+0000 7f8f0debe700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f080bd970 con 0x7f8f08063480 2023-10-18T16:07:25.753 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.745+0000 7f8f0debe700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f080be7d0 con 0x7f8f0805e050 2023-10-18T16:07:25.754 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.745+0000 7f8f0debe700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f080c0410 con 0x7f8f080bf930 2023-10-18T16:07:25.754 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.745+0000 7f8f077fe700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8f080bf930 0x7f8f08066cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:25.754 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.745+0000 7f8f077fe700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8f080bf930 0x7f8f08066cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:58474/0 (socket says 172.21.15.151:58474) 2023-10-18T16:07:25.754 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.745+0000 7f8f077fe700 1 -- 172.21.15.151:0/1933519110 learned_addr learned my addr 172.21.15.151:0/1933519110 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-10-18T16:07:25.755 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.745+0000 7f8f06ffd700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f8f0805e050 0x7f8f08062d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:58254/0 (socket says 172.21.15.151:58254) 2023-10-18T16:07:25.755 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.746+0000 7f8f077fe700 1 -- 172.21.15.151:0/1933519110 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8f080bf930 msgr2=0x7f8f08066cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-18T16:07:25.755 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.746+0000 7f8f077fe700 1 -- 172.21.15.151:0/1933519110 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8f080bf930 msgr2=0x7f8f08066cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-18T16:07:25.755 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.746+0000 7f8f077fe700 1 --2- 172.21.15.151:0/1933519110 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8f080bf930 0x7f8f08066cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-10-18T16:07:25.756 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.746+0000 7f8f077fe700 1 --2- 172.21.15.151:0/1933519110 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8f080bf930 0x7f8f08066cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:07:25.756 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.746+0000 7f8f067fc700 1 -- 172.21.15.151:0/1933519110 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 498434081 0 0) 0x7f8f080be7d0 con 0x7f8f0805e050 2023-10-18T16:07:25.756 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.746+0000 7f8f067fc700 1 -- 172.21.15.151:0/1933519110 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ef00033f0 con 0x7f8f0805e050 2023-10-18T16:07:25.756 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.746+0000 7f8f067fc700 1 -- 172.21.15.151:0/1933519110 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 967158054 0 0) 0x7f8f080bd970 con 0x7f8f08063480 2023-10-18T16:07:25.756 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.746+0000 7f8f067fc700 1 -- 172.21.15.151:0/1933519110 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f080be7d0 con 0x7f8f08063480 2023-10-18T16:07:25.757 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.746+0000 7f8f067fc700 1 -- 172.21.15.151:0/1933519110 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 525773633 0 0) 0x7f8ef00033f0 con 0x7f8f0805e050 2023-10-18T16:07:25.757 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.746+0000 7f8f067fc700 1 -- 172.21.15.151:0/1933519110 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8f080bf930 msgr2=0x7f8f08066cc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:07:25.757 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.746+0000 7f8f067fc700 1 --2- 172.21.15.151:0/1933519110 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8f080bf930 0x7f8f08066cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:25.757 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.746+0000 7f8f067fc700 1 -- 172.21.15.151:0/1933519110 >> v1:172.21.15.88:6789/0 conn(0x7f8f08063480 legacy=0x7f8f0814d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:25.758 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.746+0000 7f8f067fc700 1 -- 172.21.15.151:0/1933519110 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f080bdbc0 con 0x7f8f0805e050 2023-10-18T16:07:25.758 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.746+0000 7f8f067fc700 1 -- 172.21.15.151:0/1933519110 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441715075 0 0) 0x7f8efc001470 con 0x7f8f0805e050 2023-10-18T16:07:25.758 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.746+0000 7f8f067fc700 1 -- 172.21.15.151:0/1933519110 >> v1:172.21.15.151:6789/0 conn(0x7f8f0805e050 legacy=0x7f8f08062d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:25.758 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.746+0000 7f8f067fc700 1 --2- 172.21.15.151:0/1933519110 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8f080bf930 0x7f8ef0003a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:25.758 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.746+0000 7f8f067fc700 1 --2- 172.21.15.151:0/1933519110 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8ef0003f80 0x7f8ef0004690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:25.759 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.746+0000 7f8f067fc700 1 --2- 172.21.15.151:0/1933519110 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8ef0004b80 0x7f8ef0005000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:25.759 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.746+0000 7f8f067fc700 1 -- 172.21.15.151:0/1933519110 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f080c0410 con 0x7f8ef0004b80 2023-10-18T16:07:25.759 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.746+0000 7f8f067fc700 1 -- 172.21.15.151:0/1933519110 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef000b9f0 con 0x7f8ef0003f80 2023-10-18T16:07:25.759 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.746+0000 7f8f067fc700 1 -- 172.21.15.151:0/1933519110 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef000bbb0 con 0x7f8f080bf930 2023-10-18T16:07:25.759 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.746+0000 7f8f077fe700 1 --2- 172.21.15.151:0/1933519110 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8f080bf930 0x7f8ef0003a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:25.760 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.746+0000 7f8f06ffd700 1 --2- 172.21.15.151:0/1933519110 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8ef0003f80 0x7f8ef0004690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:25.760 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.746+0000 7f8f077fe700 1 -- 172.21.15.151:0/1933519110 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8f080bf930 msgr2=0x7f8ef0003a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-18T16:07:25.760 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.746+0000 7f8f077fe700 1 -- 172.21.15.151:0/1933519110 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8f080bf930 msgr2=0x7f8ef0003a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-18T16:07:25.760 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.746+0000 7f8f077fe700 1 --2- 172.21.15.151:0/1933519110 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8f080bf930 0x7f8ef0003a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-10-18T16:07:25.760 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.746+0000 7f8f07fff700 1 --2- 172.21.15.151:0/1933519110 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8ef0004b80 0x7f8ef0005000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:25.761 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.746+0000 7f8f077fe700 1 --2- 172.21.15.151:0/1933519110 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8f080bf930 0x7f8ef0003a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:07:25.761 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.746+0000 7f8f06ffd700 1 -- 172.21.15.151:0/1933519110 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8f080bf930 msgr2=0x7f8ef0003a70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:07:25.761 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.746+0000 7f8f06ffd700 1 --2- 172.21.15.151:0/1933519110 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8f080bf930 0x7f8ef0003a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:25.761 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.746+0000 7f8f06ffd700 1 -- 172.21.15.151:0/1933519110 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8ef0004b80 msgr2=0x7f8ef0005000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:25.761 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.746+0000 7f8f06ffd700 1 --2- 172.21.15.151:0/1933519110 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8ef0004b80 0x7f8ef0005000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:25.762 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.746+0000 7f8f06ffd700 1 -- 172.21.15.151:0/1933519110 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ef00033f0 con 0x7f8ef0003f80 2023-10-18T16:07:25.762 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.747+0000 7f8f06ffd700 1 --2- 172.21.15.151:0/1933519110 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8ef0003f80 0x7f8ef0004690 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f8f08069f70 tx=0x7f8efc001e30).ready entity=mon.1 client_cookie=e0a61ffbf266a9c4 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:25.762 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.747+0000 7f8f067fc700 1 -- 172.21.15.151:0/1933519110 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8efc0070d0 con 0x7f8ef0003f80 2023-10-18T16:07:25.762 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.747+0000 7f8f067fc700 1 -- 172.21.15.151:0/1933519110 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8efc007230 con 0x7f8ef0003f80 2023-10-18T16:07:25.762 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.747+0000 7f8f067fc700 1 -- 172.21.15.151:0/1933519110 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8efc0075d0 con 0x7f8ef0003f80 2023-10-18T16:07:25.763 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.747+0000 7f8f0debe700 1 -- 172.21.15.151:0/1933519110 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8ef0003f80 msgr2=0x7f8ef0004690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:25.763 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.747+0000 7f8f0debe700 1 --2- 172.21.15.151:0/1933519110 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8ef0003f80 0x7f8ef0004690 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f8f08069f70 tx=0x7f8efc001e30).stop 2023-10-18T16:07:25.763 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.747+0000 7f8f07fff700 1 -- 172.21.15.151:0/1933519110 reap_dead start 2023-10-18T16:07:25.763 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.747+0000 7f8f0debe700 1 -- 172.21.15.151:0/1933519110 shutdown_connections 2023-10-18T16:07:25.763 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.747+0000 7f8f0debe700 1 -- 172.21.15.151:0/1933519110 >> 172.21.15.151:0/1933519110 conn(0x7f8f08064f00 msgr2=0x7f8f08065310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:25.764 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.747+0000 7f8f0debe700 1 -- 172.21.15.151:0/1933519110 shutdown_connections 2023-10-18T16:07:25.764 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.747+0000 7f8f0debe700 1 -- 172.21.15.151:0/1933519110 wait complete. 2023-10-18T16:07:25.764 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.749+0000 7f8f0debe700 1 Processor -- start 2023-10-18T16:07:25.764 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.749+0000 7f8f0debe700 1 -- start start 2023-10-18T16:07:25.767 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.750+0000 7f8f0debe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8ef0003f80 0x7f8f08154320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:25.767 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.750+0000 7f8f0debe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f080b6130 0x7f8f08152f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:25.767 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.750+0000 7f8f0debe700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8f081533f0 0x7f8f081483e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:25.768 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.750+0000 7f8f0debe700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f080b3f60 con 0x7f8f080b6130 2023-10-18T16:07:25.768 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.750+0000 7f8f0debe700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f08153f10 con 0x7f8ef0003f80 2023-10-18T16:07:25.768 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.750+0000 7f8f0debe700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f08154b80 con 0x7f8f081533f0 2023-10-18T16:07:25.768 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.750+0000 7f8f077fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8ef0003f80 0x7f8f08154320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:25.768 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.750+0000 7f8f07fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8f081533f0 0x7f8f081483e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:25.769 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.750+0000 7f8f077fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8ef0003f80 0x7f8f08154320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:57320/0 (socket says 172.21.15.151:57320) 2023-10-18T16:07:25.769 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.750+0000 7f8f06ffd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f080b6130 0x7f8f08152f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:25.769 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.750+0000 7f8f077fe700 1 -- 172.21.15.151:0/1129040979 learned_addr learned my addr 172.21.15.151:0/1129040979 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-10-18T16:07:25.769 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.751+0000 7f8f07fff700 1 -- 172.21.15.151:0/1129040979 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8f081533f0 msgr2=0x7f8f081483e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2023-10-18T16:07:25.769 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.751+0000 7f8f07fff700 1 -- 172.21.15.151:0/1129040979 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8f081533f0 msgr2=0x7f8f081483e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-18T16:07:25.770 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.751+0000 7f8f07fff700 1 --2- 172.21.15.151:0/1129040979 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8f081533f0 0x7f8f081483e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-10-18T16:07:25.770 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.751+0000 7f8f07fff700 1 --2- 172.21.15.151:0/1129040979 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8f081533f0 0x7f8f081483e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:07:25.770 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.751+0000 7f8f077fe700 1 -- 172.21.15.151:0/1129040979 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8f081533f0 msgr2=0x7f8f081483e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-18T16:07:25.770 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.751+0000 7f8f077fe700 1 --2- 172.21.15.151:0/1129040979 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8f081533f0 0x7f8f081483e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:25.771 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.751+0000 7f8f077fe700 1 -- 172.21.15.151:0/1129040979 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f080b6130 msgr2=0x7f8f08152f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:25.771 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.751+0000 7f8f077fe700 1 --2- 172.21.15.151:0/1129040979 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f080b6130 0x7f8f08152f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:25.771 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.751+0000 7f8f077fe700 1 -- 172.21.15.151:0/1129040979 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f08154e40 con 0x7f8ef0003f80 2023-10-18T16:07:25.771 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.751+0000 7f8f06ffd700 1 --2- 172.21.15.151:0/1129040979 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f080b6130 0x7f8f08152f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2023-10-18T16:07:25.771 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.751+0000 7f8f077fe700 1 --2- 172.21.15.151:0/1129040979 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8ef0003f80 0x7f8f08154320 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f8f00007ca0 tx=0x7f8f0000b040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:25.772 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.751+0000 7f8eef7fe700 1 -- 172.21.15.151:0/1129040979 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f0000fa30 con 0x7f8ef0003f80 2023-10-18T16:07:25.772 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.751+0000 7f8eef7fe700 1 -- 172.21.15.151:0/1129040979 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f0000fb90 con 0x7f8ef0003f80 2023-10-18T16:07:25.772 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.751+0000 7f8f0debe700 1 -- 172.21.15.151:0/1129040979 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f080b32e0 con 0x7f8ef0003f80 2023-10-18T16:07:25.772 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.751+0000 7f8eef7fe700 1 -- 172.21.15.151:0/1129040979 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f00014410 con 0x7f8ef0003f80 2023-10-18T16:07:25.772 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.751+0000 7f8f0debe700 1 -- 172.21.15.151:0/1129040979 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f080b38a0 con 0x7f8ef0003f80 2023-10-18T16:07:25.773 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.751+0000 7f8eef7fe700 1 -- 172.21.15.151:0/1129040979 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 1) v1 ==== 495+0+0 (secure 0 0 0) 0x7f8f00014840 con 0x7f8ef0003f80 2023-10-18T16:07:25.773 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.751+0000 7f8eef7fe700 1 -- 172.21.15.151:0/1129040979 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1444+0+0 (secure 0 0 0) 0x7f8f0001ea80 con 0x7f8ef0003f80 2023-10-18T16:07:25.773 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.752+0000 7f8f0debe700 1 -- 172.21.15.151:0/1129040979 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f080b40a0 con 0x7f8ef0003f80 2023-10-18T16:07:25.773 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.755+0000 7f8eef7fe700 1 -- 172.21.15.151:0/1129040979 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73265 (secure 0 0 0) 0x7f8f080b40a0 con 0x7f8ef0003f80 2023-10-18T16:07:25.847 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.843+0000 7f8f0debe700 1 -- 172.21.15.151:0/1129040979 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd new", "uuid": "7af01f9c-90db-45fd-93c8-09517c9b0881", "id": 6} v 0) v1 -- 0x7f8f080672a0 con 0x7f8ef0003f80 2023-10-18T16:07:25.853 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-10-18T16:07:25.849+0000 7f07f6041680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-18T16:07:25.853 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.849+0000 7f8eef7fe700 1 -- 172.21.15.151:0/1129040979 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7af01f9c-90db-45fd-93c8-09517c9b0881", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f8f00014cd0 con 0x7f8ef0003f80 2023-10-18T16:07:25.854 INFO:teuthology.orchestra.run.smithi151.stdout:6 2023-10-18T16:07:25.855 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.850+0000 7f8f0debe700 1 -- 172.21.15.151:0/1129040979 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8ef0003f80 msgr2=0x7f8f08154320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:25.855 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.850+0000 7f8f0debe700 1 --2- 172.21.15.151:0/1129040979 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8ef0003f80 0x7f8f08154320 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f8f00007ca0 tx=0x7f8f0000b040).stop 2023-10-18T16:07:25.855 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.850+0000 7f8f0debe700 1 -- 172.21.15.151:0/1129040979 shutdown_connections 2023-10-18T16:07:25.856 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.850+0000 7f8f0debe700 1 --2- 172.21.15.151:0/1129040979 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8f081533f0 0x7f8f081483e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:25.856 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.850+0000 7f8f0debe700 1 --2- 172.21.15.151:0/1129040979 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8ef0003f80 0x7f8f08154320 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:25.856 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.850+0000 7f8f0debe700 1 --2- 172.21.15.151:0/1129040979 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f080b6130 0x7f8f08152f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:25.856 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.850+0000 7f8f0debe700 1 -- 172.21.15.151:0/1129040979 >> 172.21.15.151:0/1129040979 conn(0x7f8f08064f00 msgr2=0x7f8f080bbe50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:25.856 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.850+0000 7f8f0debe700 1 -- 172.21.15.151:0/1129040979 shutdown_connections 2023-10-18T16:07:25.857 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.850+0000 7f8f0debe700 1 -- 172.21.15.151:0/1129040979 wait complete. 2023-10-18T16:07:25.865 DEBUG:teuthology.orchestra.run.smithi151:> sudo ceph --cluster ceph osd new b527b0a5-4249-4854-a30f-bff897757239 7 2023-10-18T16:07:25.922 INFO:tasks.ceph.mgr.x.smithi151.stderr:2023-10-18T16:07:25.919+0000 7f8d9db54680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-18T16:07:25.958 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-10-18T16:07:25.954+0000 7f07f6041680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-18T16:07:25.977 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.974+0000 7fca45095700 1 Processor -- start 2023-10-18T16:07:25.978 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.974+0000 7fca45095700 1 -- start start 2023-10-18T16:07:25.980 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.975+0000 7fca45095700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fca400bbb50 0x7fca400c1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:25.980 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.975+0000 7fca45095700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca400c3030 con 0x7fca400bbf30 2023-10-18T16:07:25.981 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.975+0000 7fca45095700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca400c2140 con 0x7fca400c71b0 2023-10-18T16:07:25.981 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.975+0000 7fca45095700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca400c2420 con 0x7fca400bbb50 2023-10-18T16:07:25.981 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.975+0000 7fca3ed9d700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fca400bbb50 0x7fca400c1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:25.981 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.975+0000 7fca3e59c700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fca400c71b0 0x7fca4014d2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:58270/0 (socket says 172.21.15.151:58270) 2023-10-18T16:07:25.981 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.975+0000 7fca3e59c700 1 -- 172.21.15.151:0/453811465 learned_addr learned my addr 172.21.15.151:0/453811465 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-10-18T16:07:25.982 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.975+0000 7fca3ed9d700 1 -- 172.21.15.151:0/453811465 >> v1:172.21.15.151:6789/0 conn(0x7fca400c71b0 legacy=0x7fca4014d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:25.982 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.975+0000 7fca3ed9d700 1 -- 172.21.15.151:0/453811465 >> v1:172.21.15.88:6789/0 conn(0x7fca400bbf30 legacy=0x7fca40063a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:25.982 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.975+0000 7fca3ed9d700 1 -- 172.21.15.151:0/453811465 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca400c2700 con 0x7fca400bbb50 2023-10-18T16:07:25.982 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.975+0000 7fca3ed9d700 1 --2- 172.21.15.151:0/453811465 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fca400bbb50 0x7fca400c1bb0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fca34009760 tx=0x7fca34009060).ready entity=mon.1 client_cookie=55c4c83ffd5c0113 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:25.983 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.975+0000 7fca3dd9b700 1 -- 172.21.15.151:0/453811465 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca3400b610 con 0x7fca400bbb50 2023-10-18T16:07:25.983 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.975+0000 7fca3dd9b700 1 -- 172.21.15.151:0/453811465 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca3400b770 con 0x7fca400bbb50 2023-10-18T16:07:25.983 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.975+0000 7fca3dd9b700 1 -- 172.21.15.151:0/453811465 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca3400baf0 con 0x7fca400bbb50 2023-10-18T16:07:25.983 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.976+0000 7fca45095700 1 -- 172.21.15.151:0/453811465 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fca400bbb50 msgr2=0x7fca400c1bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:25.983 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.976+0000 7fca45095700 1 --2- 172.21.15.151:0/453811465 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fca400bbb50 0x7fca400c1bb0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fca34009760 tx=0x7fca34009060).stop 2023-10-18T16:07:25.984 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.976+0000 7fca45095700 1 -- 172.21.15.151:0/453811465 shutdown_connections 2023-10-18T16:07:25.984 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.976+0000 7fca45095700 1 --2- 172.21.15.151:0/453811465 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fca400bbb50 0x7fca400c1bb0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:25.984 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.976+0000 7fca45095700 1 -- 172.21.15.151:0/453811465 >> 172.21.15.151:0/453811465 conn(0x7fca4006a810 msgr2=0x7fca400c6a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:25.984 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.976+0000 7fca45095700 1 -- 172.21.15.151:0/453811465 shutdown_connections 2023-10-18T16:07:25.984 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.976+0000 7fca45095700 1 -- 172.21.15.151:0/453811465 wait complete. 2023-10-18T16:07:25.987 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.977+0000 7fca45095700 1 Processor -- start 2023-10-18T16:07:25.987 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.977+0000 7fca45095700 1 -- start start 2023-10-18T16:07:25.987 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.978+0000 7fca45095700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fca400bbb50 0x7fca400c0010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:25.988 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.978+0000 7fca45095700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca400bbf30 0x7fca40160ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:25.988 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.978+0000 7fca45095700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fca400c71b0 0x7fca400ac4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:25.988 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.978+0000 7fca45095700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca400bf760 con 0x7fca400bbf30 2023-10-18T16:07:25.988 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.978+0000 7fca45095700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca400c0500 con 0x7fca400c71b0 2023-10-18T16:07:25.988 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.978+0000 7fca45095700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca40160fa0 con 0x7fca400bbb50 2023-10-18T16:07:25.989 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.978+0000 7fca3ed9d700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fca400bbb50 0x7fca400c0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:25.989 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.978+0000 7fca3f59e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fca400c71b0 0x7fca400ac4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:25.989 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.978+0000 7fca3ed9d700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fca400bbb50 0x7fca400c0010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:58496/0 (socket says 172.21.15.151:58496) 2023-10-18T16:07:25.989 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.978+0000 7fca3ed9d700 1 -- 172.21.15.151:0/916282198 learned_addr learned my addr 172.21.15.151:0/916282198 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-10-18T16:07:25.989 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.978+0000 7fca3ed9d700 1 -- 172.21.15.151:0/916282198 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fca400bbb50 msgr2=0x7fca400c0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2023-10-18T16:07:25.989 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.978+0000 7fca3ed9d700 1 -- 172.21.15.151:0/916282198 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fca400bbb50 msgr2=0x7fca400c0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-18T16:07:25.990 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.978+0000 7fca3e59c700 1 --2- 172.21.15.151:0/916282198 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca400bbf30 0x7fca40160ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:25.990 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.978+0000 7fca3ed9d700 1 --2- 172.21.15.151:0/916282198 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fca400bbb50 0x7fca400c0010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-10-18T16:07:25.990 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.978+0000 7fca3ed9d700 1 --2- 172.21.15.151:0/916282198 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fca400bbb50 0x7fca400c0010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:07:25.990 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.978+0000 7fca3f59e700 1 -- 172.21.15.151:0/916282198 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fca400bbb50 msgr2=0x7fca400c0010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-18T16:07:25.991 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.978+0000 7fca3f59e700 1 --2- 172.21.15.151:0/916282198 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fca400bbb50 0x7fca400c0010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:25.991 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.978+0000 7fca3f59e700 1 -- 172.21.15.151:0/916282198 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca400bbf30 msgr2=0x7fca40160ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:25.991 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.978+0000 7fca3f59e700 1 --2- 172.21.15.151:0/916282198 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca400bbf30 0x7fca40160ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:25.991 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.978+0000 7fca3f59e700 1 -- 172.21.15.151:0/916282198 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca340092a0 con 0x7fca400c71b0 2023-10-18T16:07:25.991 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.979+0000 7fca3f59e700 1 --2- 172.21.15.151:0/916282198 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fca400c71b0 0x7fca400ac4a0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fca3800c790 tx=0x7fca38001530).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:25.991 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.979+0000 7fca26ffd700 1 -- 172.21.15.151:0/916282198 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca3800e4a0 con 0x7fca400c71b0 2023-10-18T16:07:25.992 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.979+0000 7fca26ffd700 1 -- 172.21.15.151:0/916282198 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca3800e600 con 0x7fca400c71b0 2023-10-18T16:07:25.992 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.979+0000 7fca45095700 1 -- 172.21.15.151:0/916282198 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca400ac010 con 0x7fca400c71b0 2023-10-18T16:07:25.992 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.979+0000 7fca26ffd700 1 -- 172.21.15.151:0/916282198 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca3800e980 con 0x7fca400c71b0 2023-10-18T16:07:25.992 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.979+0000 7fca45095700 1 -- 172.21.15.151:0/916282198 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca400c06e0 con 0x7fca400c71b0 2023-10-18T16:07:25.992 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.979+0000 7fca26ffd700 1 -- 172.21.15.151:0/916282198 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 1) v1 ==== 495+0+0 (secure 0 0 0) 0x7fca3801b3f0 con 0x7fca400c71b0 2023-10-18T16:07:25.993 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.979+0000 7fca26ffd700 1 -- 172.21.15.151:0/916282198 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1559+0+0 (secure 0 0 0) 0x7fca3800ed70 con 0x7fca400c71b0 2023-10-18T16:07:25.993 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.980+0000 7fca45095700 1 -- 172.21.15.151:0/916282198 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca4006c4e0 con 0x7fca400c71b0 2023-10-18T16:07:25.993 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:25.982+0000 7fca26ffd700 1 -- 172.21.15.151:0/916282198 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73265 (secure 0 0 0) 0x7fca38020060 con 0x7fca400c71b0 2023-10-18T16:07:26.027 INFO:tasks.ceph.mgr.x.smithi151.stderr:2023-10-18T16:07:26.024+0000 7f8d9db54680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-18T16:07:26.066 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:26.062+0000 7fca45095700 1 -- 172.21.15.151:0/916282198 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd new", "uuid": "b527b0a5-4249-4854-a30f-bff897757239", "id": 7} v 0) v1 -- 0x7fca400c29e0 con 0x7fca400c71b0 2023-10-18T16:07:26.071 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:26.067+0000 7fca26ffd700 1 -- 172.21.15.151:0/916282198 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b527b0a5-4249-4854-a30f-bff897757239", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7fca3801b6a0 con 0x7fca400c71b0 2023-10-18T16:07:26.071 INFO:teuthology.orchestra.run.smithi151.stdout:7 2023-10-18T16:07:26.072 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:26.068+0000 7fca45095700 1 -- 172.21.15.151:0/916282198 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fca400c71b0 msgr2=0x7fca400ac4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:26.073 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:26.068+0000 7fca45095700 1 --2- 172.21.15.151:0/916282198 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fca400c71b0 0x7fca400ac4a0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fca3800c790 tx=0x7fca38001530).stop 2023-10-18T16:07:26.073 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:26.068+0000 7fca45095700 1 -- 172.21.15.151:0/916282198 shutdown_connections 2023-10-18T16:07:26.073 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:26.068+0000 7fca45095700 1 --2- 172.21.15.151:0/916282198 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fca400c71b0 0x7fca400ac4a0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:26.073 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:26.068+0000 7fca45095700 1 --2- 172.21.15.151:0/916282198 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fca400bbb50 0x7fca400c0010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:26.073 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:26.068+0000 7fca45095700 1 --2- 172.21.15.151:0/916282198 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca400bbf30 0x7fca40160ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:26.074 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:26.068+0000 7fca45095700 1 -- 172.21.15.151:0/916282198 >> 172.21.15.151:0/916282198 conn(0x7fca4006a810 msgr2=0x7fca40063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:26.074 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:26.068+0000 7fca45095700 1 -- 172.21.15.151:0/916282198 shutdown_connections 2023-10-18T16:07:26.074 INFO:teuthology.orchestra.run.smithi151.stderr:2023-10-18T16:07:26.068+0000 7fca45095700 1 -- 172.21.15.151:0/916282198 wait complete. 2023-10-18T16:07:26.084 INFO:tasks.ceph.osd.0:Restarting daemon 2023-10-18T16:07:26.084 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-10-18T16:07:26.086 INFO:tasks.ceph.osd.0:Started 2023-10-18T16:07:26.087 INFO:tasks.ceph.osd.1:Restarting daemon 2023-10-18T16:07:26.087 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-10-18T16:07:26.089 INFO:tasks.ceph.osd.1:Started 2023-10-18T16:07:26.090 INFO:tasks.ceph.osd.2:Restarting daemon 2023-10-18T16:07:26.090 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-10-18T16:07:26.092 INFO:tasks.ceph.osd.2:Started 2023-10-18T16:07:26.092 INFO:tasks.ceph.osd.3:Restarting daemon 2023-10-18T16:07:26.093 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-10-18T16:07:26.095 INFO:tasks.ceph.osd.3:Started 2023-10-18T16:07:26.095 INFO:tasks.ceph.osd.4:Restarting daemon 2023-10-18T16:07:26.095 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-10-18T16:07:26.098 INFO:tasks.ceph.osd.4:Started 2023-10-18T16:07:26.098 INFO:tasks.ceph.osd.5:Restarting daemon 2023-10-18T16:07:26.099 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-10-18T16:07:26.101 INFO:tasks.ceph.osd.5:Started 2023-10-18T16:07:26.101 INFO:tasks.ceph.osd.6:Restarting daemon 2023-10-18T16:07:26.101 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-10-18T16:07:26.104 INFO:tasks.ceph.osd.6:Started 2023-10-18T16:07:26.104 INFO:tasks.ceph.osd.7:Restarting daemon 2023-10-18T16:07:26.104 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-10-18T16:07:26.107 INFO:tasks.ceph.osd.7:Started 2023-10-18T16:07:26.107 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-18T16:07:26.127 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-10-18T16:07:26.124+0000 7f07f6041680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-18T16:07:26.195 INFO:tasks.ceph.mgr.x.smithi151.stderr:2023-10-18T16:07:26.192+0000 7f8d9db54680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-18T16:07:26.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.260+0000 7fc95359e700 1 Processor -- start 2023-10-18T16:07:26.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.260+0000 7fc95359e700 1 -- start start 2023-10-18T16:07:26.264 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-10-18T16:07:26.260+0000 7f07f6041680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-18T16:07:26.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.261+0000 7fc95359e700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc954064360 0x7fc9541538a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:26.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.261+0000 7fc95359e700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fc95414dba0 0x7fc954153d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:26.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.261+0000 7fc95359e700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc954151840 con 0x7fc95414d7c0 2023-10-18T16:07:26.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.261+0000 7fc95359e700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9540c42d0 con 0x7fc954064360 2023-10-18T16:07:26.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.261+0000 7fc95359e700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc954064830 con 0x7fc95414dba0 2023-10-18T16:07:26.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.261+0000 7fc951d9b700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc954064360 0x7fc9541538a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:26.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.261+0000 7fc95259c700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fc95414d7c0 0x7fc9540a8de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:37932/0 (socket says 172.21.15.88:37932) 2023-10-18T16:07:26.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.261+0000 7fc95259c700 1 -- 172.21.15.88:0/3159961646 learned_addr learned my addr 172.21.15.88:0/3159961646 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:07:26.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.261+0000 7fc952d9d700 1 --2- 172.21.15.88:0/3159961646 >> v2:172.21.15.151:3300/0 conn(0x7fc95414dba0 0x7fc954153d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:26.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.261+0000 7fc95159a700 1 -- 172.21.15.88:0/3159961646 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1526484922 0 0) 0x7fc954151840 con 0x7fc95414d7c0 2023-10-18T16:07:26.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.261+0000 7fc95159a700 1 -- 172.21.15.88:0/3159961646 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc93c0033f0 con 0x7fc95414d7c0 2023-10-18T16:07:26.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.261+0000 7fc951d9b700 1 -- 172.21.15.88:0/3159961646 >> v2:172.21.15.151:3300/0 conn(0x7fc95414dba0 msgr2=0x7fc954153d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:26.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.261+0000 7fc951d9b700 1 --2- 172.21.15.88:0/3159961646 >> v2:172.21.15.151:3300/0 conn(0x7fc95414dba0 0x7fc954153d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:26.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.261+0000 7fc951d9b700 1 -- 172.21.15.88:0/3159961646 >> v1:172.21.15.88:6789/0 conn(0x7fc95414d7c0 legacy=0x7fc9540a8de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:26.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.261+0000 7fc951d9b700 1 -- 172.21.15.88:0/3159961646 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc954064b10 con 0x7fc954064360 2023-10-18T16:07:26.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.261+0000 7fc951d9b700 1 --2- 172.21.15.88:0/3159961646 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc954064360 0x7fc9541538a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fc954065940 tx=0x7fc948009060).ready entity=mon.0 client_cookie=bc72c93ab62a2a8c server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:26.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.261+0000 7fc95159a700 1 -- 172.21.15.88:0/3159961646 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc94800d4a0 con 0x7fc954064360 2023-10-18T16:07:26.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.261+0000 7fc95159a700 1 -- 172.21.15.88:0/3159961646 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc94800d600 con 0x7fc954064360 2023-10-18T16:07:26.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.262+0000 7fc95159a700 1 -- 172.21.15.88:0/3159961646 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc94800d980 con 0x7fc954064360 2023-10-18T16:07:26.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.262+0000 7fc95359e700 1 -- 172.21.15.88:0/3159961646 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc954064360 msgr2=0x7fc9541538a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:26.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.262+0000 7fc95359e700 1 --2- 172.21.15.88:0/3159961646 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc954064360 0x7fc9541538a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fc954065940 tx=0x7fc948009060).stop 2023-10-18T16:07:26.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.262+0000 7fc95359e700 1 -- 172.21.15.88:0/3159961646 shutdown_connections 2023-10-18T16:07:26.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.262+0000 7fc95359e700 1 --2- 172.21.15.88:0/3159961646 >> v2:172.21.15.151:3300/0 conn(0x7fc95414dba0 0x7fc954153d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:26.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.262+0000 7fc95359e700 1 --2- 172.21.15.88:0/3159961646 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc954064360 0x7fc9541538a0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:26.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.262+0000 7fc95359e700 1 -- 172.21.15.88:0/3159961646 >> 172.21.15.88:0/3159961646 conn(0x7fc954069d80 msgr2=0x7fc95406a190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:26.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.262+0000 7fc95359e700 1 -- 172.21.15.88:0/3159961646 shutdown_connections 2023-10-18T16:07:26.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.262+0000 7fc95359e700 1 -- 172.21.15.88:0/3159961646 wait complete. 2023-10-18T16:07:26.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.263+0000 7fc95359e700 1 Processor -- start 2023-10-18T16:07:26.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.263+0000 7fc95359e700 1 -- start start 2023-10-18T16:07:26.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.264+0000 7fc95359e700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc954064360 0x7fc954149130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:26.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.264+0000 7fc95359e700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc95414d7c0 0x7fc954163980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:26.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.264+0000 7fc95359e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc95414dba0 0x7fc9540c2a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:26.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.264+0000 7fc95359e700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc954149620 con 0x7fc954064360 2023-10-18T16:07:26.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.264+0000 7fc95359e700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9540aa6e0 con 0x7fc95414dba0 2023-10-18T16:07:26.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.264+0000 7fc95359e700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9540a9c50 con 0x7fc95414d7c0 2023-10-18T16:07:26.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.264+0000 7fc95259c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc954064360 0x7fc954149130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:26.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.264+0000 7fc95259c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc954064360 0x7fc954149130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56948/0 (socket says 172.21.15.88:56948) 2023-10-18T16:07:26.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.264+0000 7fc95259c700 1 -- 172.21.15.88:0/2969887349 learned_addr learned my addr 172.21.15.88:0/2969887349 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:26.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.264+0000 7fc951d9b700 1 --2- 172.21.15.88:0/2969887349 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc95414d7c0 0x7fc954163980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:26.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.264+0000 7fc952d9d700 1 --2- 172.21.15.88:0/2969887349 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc95414dba0 0x7fc9540c2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:26.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.264+0000 7fc951d9b700 1 -- 172.21.15.88:0/2969887349 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc95414d7c0 msgr2=0x7fc954163980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-10-18T16:07:26.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.264+0000 7fc951d9b700 1 -- 172.21.15.88:0/2969887349 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc95414d7c0 msgr2=0x7fc954163980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-18T16:07:26.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.264+0000 7fc951d9b700 1 --2- 172.21.15.88:0/2969887349 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc95414d7c0 0x7fc954163980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-10-18T16:07:26.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.264+0000 7fc951d9b700 1 --2- 172.21.15.88:0/2969887349 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc95414d7c0 0x7fc954163980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:07:26.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.264+0000 7fc95259c700 1 -- 172.21.15.88:0/2969887349 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc95414d7c0 msgr2=0x7fc954163980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-18T16:07:26.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.264+0000 7fc95259c700 1 --2- 172.21.15.88:0/2969887349 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc95414d7c0 0x7fc954163980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:26.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.264+0000 7fc95259c700 1 -- 172.21.15.88:0/2969887349 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc95414dba0 msgr2=0x7fc9540c2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:26.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.264+0000 7fc95259c700 1 --2- 172.21.15.88:0/2969887349 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc95414dba0 0x7fc9540c2a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:26.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.264+0000 7fc95259c700 1 -- 172.21.15.88:0/2969887349 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9480092a0 con 0x7fc954064360 2023-10-18T16:07:26.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.265+0000 7fc95259c700 1 --2- 172.21.15.88:0/2969887349 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc954064360 0x7fc954149130 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fc94400c5c0 tx=0x7fc944007c80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:26.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.265+0000 7fc9427fc700 1 -- 172.21.15.88:0/2969887349 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc944007e40 con 0x7fc954064360 2023-10-18T16:07:26.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.265+0000 7fc9427fc700 1 -- 172.21.15.88:0/2969887349 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9440072e0 con 0x7fc954064360 2023-10-18T16:07:26.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.265+0000 7fc95359e700 1 -- 172.21.15.88:0/2969887349 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9540ab430 con 0x7fc954064360 2023-10-18T16:07:26.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.265+0000 7fc9427fc700 1 -- 172.21.15.88:0/2969887349 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9440057a0 con 0x7fc954064360 2023-10-18T16:07:26.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.265+0000 7fc95359e700 1 -- 172.21.15.88:0/2969887349 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9540bdfe0 con 0x7fc954064360 2023-10-18T16:07:26.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.265+0000 7fc9427fc700 1 -- 172.21.15.88:0/2969887349 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 1) v1 ==== 495+0+0 (secure 0 0 0) 0x7fc944013450 con 0x7fc954064360 2023-10-18T16:07:26.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.265+0000 7fc9427fc700 1 -- 172.21.15.88:0/2969887349 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1674+0+0 (secure 0 0 0) 0x7fc944005df0 con 0x7fc954064360 2023-10-18T16:07:26.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.266+0000 7fc95359e700 1 -- 172.21.15.88:0/2969887349 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc954083b10 con 0x7fc954064360 2023-10-18T16:07:26.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.268+0000 7fc9427fc700 1 -- 172.21.15.88:0/2969887349 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73265 (secure 0 0 0) 0x7fc944013980 con 0x7fc954064360 2023-10-18T16:07:26.280 INFO:tasks.ceph.mgr.x.smithi151.stderr:2023-10-18T16:07:26.270+0000 7f8d9db54680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-18T16:07:26.336 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-10-18T16:07:26.333+0000 7f07f6041680 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-18T16:07:26.338 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.334+0000 7fc95359e700 1 -- 172.21.15.88:0/2969887349 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc954064df0 con 0x7fc954064360 2023-10-18T16:07:26.338 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.335+0000 7fc9427fc700 1 -- 172.21.15.88:0/2969887349 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7045 (secure 0 0 0) 0x7fc944013b20 con 0x7fc954064360 2023-10-18T16:07:26.339 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:07:26.339 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":10,"fsid":"fb1f77e8-ead0-4f90-8128-fdf5137eb16f","created":"2023-10-18T16:07:23.662468+0000","modified":"2023-10-18T16:07:26.064784+0000","last_up_change":"0.000000","last_in_change":"2023-10-18T16:07:26.064784+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"e421c20c-f7bc-4574-97a2-201bb4a16b4b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9485c5ec-379a-4038-a2d8-1f1081fe2eea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"671895e2-57f3-4f1a-ba4a-a580af1f3ec2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0c4e6e7f-7d9c-4e65-beec-fd0ca384f320","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"29570691-ef35-4b80-9247-aa3703877af4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"26540b7b-032a-4b3b-bf6c-b131a449fa99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7af01f9c-90db-45fd-93c8-09517c9b0881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b527b0a5-4249-4854-a30f-bff897757239","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-18T16:07:26.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.336+0000 7fc95359e700 1 -- 172.21.15.88:0/2969887349 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc954064360 msgr2=0x7fc954149130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:26.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.336+0000 7fc95359e700 1 --2- 172.21.15.88:0/2969887349 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc954064360 0x7fc954149130 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fc94400c5c0 tx=0x7fc944007c80).stop 2023-10-18T16:07:26.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.336+0000 7fc95359e700 1 -- 172.21.15.88:0/2969887349 shutdown_connections 2023-10-18T16:07:26.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.336+0000 7fc95359e700 1 --2- 172.21.15.88:0/2969887349 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc95414dba0 0x7fc9540c2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:26.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.336+0000 7fc95359e700 1 --2- 172.21.15.88:0/2969887349 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc954064360 0x7fc954149130 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:26.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.336+0000 7fc95359e700 1 --2- 172.21.15.88:0/2969887349 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc95414d7c0 0x7fc954163980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:26.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.336+0000 7fc95359e700 1 -- 172.21.15.88:0/2969887349 >> 172.21.15.88:0/2969887349 conn(0x7fc954069d80 msgr2=0x7fc95406a160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:26.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.336+0000 7fc95359e700 1 -- 172.21.15.88:0/2969887349 shutdown_connections 2023-10-18T16:07:26.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:26.336+0000 7fc95359e700 1 -- 172.21.15.88:0/2969887349 wait complete. 2023-10-18T16:07:26.342 INFO:tasks.ceph.mgr.x.smithi151.stderr:2023-10-18T16:07:26.338+0000 7f8d9db54680 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-18T16:07:26.351 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-10-18T16:07:26.352 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-18T16:07:26.602 INFO:tasks.ceph.mgr.x.smithi151.stderr:2023-10-18T16:07:26.598+0000 7f8d9db54680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-18T16:07:26.609 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-10-18T16:07:26.605+0000 7f07f6041680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-18T16:07:26.702 INFO:tasks.ceph.osd.4.smithi151.stderr:2023-10-18T16:07:26.698+0000 7f1d41306200 -1 Falling back to public interface 2023-10-18T16:07:26.704 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-10-18T16:07:26.700+0000 7f0a8dbe1200 -1 Falling back to public interface 2023-10-18T16:07:26.705 INFO:tasks.ceph.osd.5.smithi151.stderr:2023-10-18T16:07:26.701+0000 7f383af81200 -1 Falling back to public interface 2023-10-18T16:07:26.706 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-10-18T16:07:26.702+0000 7f955ca9c200 -1 Falling back to public interface 2023-10-18T16:07:26.711 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-10-18T16:07:26.708+0000 7fd71bb78200 -1 Falling back to public interface 2023-10-18T16:07:26.715 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-10-18T16:07:26.712+0000 7f25e92d3200 -1 Falling back to public interface 2023-10-18T16:07:26.718 INFO:tasks.ceph.osd.6.smithi151.stderr:2023-10-18T16:07:26.714+0000 7f8145eb5200 -1 Falling back to public interface 2023-10-18T16:07:26.723 INFO:tasks.ceph.osd.7.smithi151.stderr:2023-10-18T16:07:26.719+0000 7f1205428200 -1 Falling back to public interface 2023-10-18T16:07:26.802 INFO:tasks.ceph.mgr.x.smithi151.stderr:2023-10-18T16:07:26.798+0000 7f8d9db54680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-18T16:07:26.809 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-10-18T16:07:26.806+0000 7f07f6041680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-18T16:07:27.157 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-18T16:07:27.263 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.260+0000 7f0bba62a700 1 Processor -- start 2023-10-18T16:07:27.264 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.260+0000 7f0bba62a700 1 -- start start 2023-10-18T16:07:27.266 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.261+0000 7f0bba62a700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bb4066be0 0x7f0bb4064770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:27.266 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.261+0000 7f0bba62a700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0bb40bfa10 0x7f0bb40c0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:27.266 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.261+0000 7f0bba62a700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bb4151680 con 0x7f0bb4064390 2023-10-18T16:07:27.267 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.261+0000 7f0bba62a700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bb40bcc10 con 0x7f0bb40bfa10 2023-10-18T16:07:27.267 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.261+0000 7f0bba62a700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb40672b0 con 0x7f0bb4066be0 2023-10-18T16:07:27.267 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.261+0000 7f0bb9628700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f0bb4064390 0x7f0bb40bd0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:37948/0 (socket says 172.21.15.88:37948) 2023-10-18T16:07:27.267 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.262+0000 7f0bb9e29700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0bb40bfa10 0x7f0bb40c0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:27.267 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.262+0000 7f0bb9628700 1 -- 172.21.15.88:0/3442964100 learned_addr learned my addr 172.21.15.88:0/3442964100 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:07:27.268 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.262+0000 7f0bb8e27700 1 --2- 172.21.15.88:0/3442964100 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bb4066be0 0x7f0bb4064770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:27.268 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.262+0000 7f0bb8e27700 1 -- 172.21.15.88:0/3442964100 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bb4066be0 msgr2=0x7f0bb4064770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-18T16:07:27.268 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.262+0000 7f0bb8e27700 1 -- 172.21.15.88:0/3442964100 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bb4066be0 msgr2=0x7f0bb4064770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-18T16:07:27.268 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.262+0000 7f0bb8e27700 1 --2- 172.21.15.88:0/3442964100 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bb4066be0 0x7f0bb4064770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-10-18T16:07:27.268 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.262+0000 7f0bb9e29700 1 -- 172.21.15.88:0/3442964100 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bb4066be0 msgr2=0x7f0bb4064770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:27.269 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.262+0000 7f0bb8e27700 1 --2- 172.21.15.88:0/3442964100 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bb4066be0 0x7f0bb4064770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:07:27.269 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.262+0000 7f0bb9e29700 1 --2- 172.21.15.88:0/3442964100 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bb4066be0 0x7f0bb4064770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:27.269 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.262+0000 7f0bb9e29700 1 -- 172.21.15.88:0/3442964100 >> v1:172.21.15.88:6789/0 conn(0x7f0bb4064390 legacy=0x7f0bb40bd0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:27.269 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.262+0000 7f0bb9e29700 1 -- 172.21.15.88:0/3442964100 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bb4067550 con 0x7f0bb40bfa10 2023-10-18T16:07:27.269 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.262+0000 7f0babfff700 1 -- 172.21.15.88:0/3442964100 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3739410877 0 0) 0x7f0bb4151680 con 0x7f0bb4064390 2023-10-18T16:07:27.270 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.262+0000 7f0bb9e29700 1 --2- 172.21.15.88:0/3442964100 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0bb40bfa10 0x7f0bb40c0640 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f0bac009e60 tx=0x7f0bac009780).ready entity=mon.0 client_cookie=888762003e4aab96 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:27.270 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.262+0000 7f0babfff700 1 -- 172.21.15.88:0/3442964100 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bac00e070 con 0x7f0bb40bfa10 2023-10-18T16:07:27.270 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.262+0000 7f0babfff700 1 -- 172.21.15.88:0/3442964100 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bac00bc30 con 0x7f0bb40bfa10 2023-10-18T16:07:27.270 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.262+0000 7f0babfff700 1 -- 172.21.15.88:0/3442964100 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bac012410 con 0x7f0bb40bfa10 2023-10-18T16:07:27.274 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.263+0000 7f0bba62a700 1 -- 172.21.15.88:0/3442964100 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0bb40bfa10 msgr2=0x7f0bb40c0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:27.274 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.263+0000 7f0bba62a700 1 --2- 172.21.15.88:0/3442964100 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0bb40bfa10 0x7f0bb40c0640 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f0bac009e60 tx=0x7f0bac009780).stop 2023-10-18T16:07:27.274 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.263+0000 7f0bba62a700 1 -- 172.21.15.88:0/3442964100 shutdown_connections 2023-10-18T16:07:27.274 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.263+0000 7f0bba62a700 1 --2- 172.21.15.88:0/3442964100 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0bb40bfa10 0x7f0bb40c0640 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:27.275 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.263+0000 7f0bba62a700 1 --2- 172.21.15.88:0/3442964100 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bb4066be0 0x7f0bb4064770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:27.275 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.263+0000 7f0bba62a700 1 -- 172.21.15.88:0/3442964100 >> 172.21.15.88:0/3442964100 conn(0x7f0bb40c3f60 msgr2=0x7f0bb40bff30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:27.275 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.263+0000 7f0bba62a700 1 -- 172.21.15.88:0/3442964100 shutdown_connections 2023-10-18T16:07:27.275 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.263+0000 7f0bba62a700 1 -- 172.21.15.88:0/3442964100 wait complete. 2023-10-18T16:07:27.275 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.265+0000 7f0bba62a700 1 Processor -- start 2023-10-18T16:07:27.276 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.265+0000 7f0bba62a700 1 -- start start 2023-10-18T16:07:27.276 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.266+0000 7f0bba62a700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0bb4064390 0x7f0bb40b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:27.276 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.266+0000 7f0bba62a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0bb4066be0 0x7f0bb414dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:27.276 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.266+0000 7f0bba62a700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bb40bfa10 0x7f0bb4161d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:27.276 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.266+0000 7f0bba62a700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb4149280 con 0x7f0bb4064390 2023-10-18T16:07:27.277 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.266+0000 7f0bba62a700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb40b8080 con 0x7f0bb4066be0 2023-10-18T16:07:27.277 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.266+0000 7f0bba62a700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb4161340 con 0x7f0bb40bfa10 2023-10-18T16:07:27.277 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.266+0000 7f0bb9628700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0bb4064390 0x7f0bb40b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:27.277 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.266+0000 7f0bb9628700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0bb4064390 0x7f0bb40b7230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56974/0 (socket says 172.21.15.88:56974) 2023-10-18T16:07:27.277 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.266+0000 7f0bb9628700 1 -- 172.21.15.88:0/1711963976 learned_addr learned my addr 172.21.15.88:0/1711963976 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:27.277 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.267+0000 7f0bb9e29700 1 --2- 172.21.15.88:0/1711963976 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bb40bfa10 0x7f0bb4161d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:27.278 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.267+0000 7f0bb8e27700 1 --2- 172.21.15.88:0/1711963976 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0bb4066be0 0x7f0bb414dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:27.278 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.267+0000 7f0bb9e29700 1 -- 172.21.15.88:0/1711963976 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bb40bfa10 msgr2=0x7f0bb4161d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2023-10-18T16:07:27.278 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.267+0000 7f0bb9e29700 1 -- 172.21.15.88:0/1711963976 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bb40bfa10 msgr2=0x7f0bb4161d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-18T16:07:27.278 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.267+0000 7f0bb9e29700 1 --2- 172.21.15.88:0/1711963976 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bb40bfa10 0x7f0bb4161d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-10-18T16:07:27.278 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.267+0000 7f0bb9e29700 1 --2- 172.21.15.88:0/1711963976 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bb40bfa10 0x7f0bb4161d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:07:27.279 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.267+0000 7f0bb9628700 1 -- 172.21.15.88:0/1711963976 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bb40bfa10 msgr2=0x7f0bb4161d50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-18T16:07:27.279 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.267+0000 7f0bb9628700 1 --2- 172.21.15.88:0/1711963976 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bb40bfa10 0x7f0bb4161d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:27.279 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.267+0000 7f0bb9628700 1 -- 172.21.15.88:0/1711963976 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0bb4066be0 msgr2=0x7f0bb414dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:27.279 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.267+0000 7f0bb9628700 1 --2- 172.21.15.88:0/1711963976 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0bb4066be0 0x7f0bb414dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:27.279 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.267+0000 7f0bb9628700 1 -- 172.21.15.88:0/1711963976 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bac009a20 con 0x7f0bb4064390 2023-10-18T16:07:27.280 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.267+0000 7f0bb8e27700 1 --2- 172.21.15.88:0/1711963976 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0bb4066be0 0x7f0bb414dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:07:27.280 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.267+0000 7f0bb9628700 1 --2- 172.21.15.88:0/1711963976 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0bb4064390 0x7f0bb40b7230 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f0bb0001eb0 tx=0x7f0bb0007e90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:27.280 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.267+0000 7f0ba97fa700 1 -- 172.21.15.88:0/1711963976 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb00072e0 con 0x7f0bb4064390 2023-10-18T16:07:27.280 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.267+0000 7f0ba97fa700 1 -- 172.21.15.88:0/1711963976 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bb0007440 con 0x7f0bb4064390 2023-10-18T16:07:27.280 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.267+0000 7f0bba62a700 1 -- 172.21.15.88:0/1711963976 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bb414f0c0 con 0x7f0bb4064390 2023-10-18T16:07:27.280 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.267+0000 7f0ba97fa700 1 -- 172.21.15.88:0/1711963976 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb00057a0 con 0x7f0bb4064390 2023-10-18T16:07:27.281 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.267+0000 7f0bba62a700 1 -- 172.21.15.88:0/1711963976 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bb41618c0 con 0x7f0bb4064390 2023-10-18T16:07:27.281 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.267+0000 7f0ba97fa700 1 -- 172.21.15.88:0/1711963976 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 1) v1 ==== 495+0+0 (secure 0 0 0) 0x7f0bb0013450 con 0x7f0bb4064390 2023-10-18T16:07:27.281 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.267+0000 7f0ba97fa700 1 -- 172.21.15.88:0/1711963976 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1674+0+0 (secure 0 0 0) 0x7f0bb0005df0 con 0x7f0bb4064390 2023-10-18T16:07:27.281 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.269+0000 7f0bba62a700 1 -- 172.21.15.88:0/1711963976 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bb40bd3b0 con 0x7f0bb4064390 2023-10-18T16:07:27.282 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.271+0000 7f0ba97fa700 1 -- 172.21.15.88:0/1711963976 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73265 (secure 0 0 0) 0x7f0bb414f0c0 con 0x7f0bb4064390 2023-10-18T16:07:27.341 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.338+0000 7f0bba62a700 1 -- 172.21.15.88:0/1711963976 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0bb4046c20 con 0x7f0bb4064390 2023-10-18T16:07:27.342 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.339+0000 7f0ba97fa700 1 -- 172.21.15.88:0/1711963976 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7045 (secure 0 0 0) 0x7f0bb0018070 con 0x7f0bb4064390 2023-10-18T16:07:27.343 INFO:teuthology.misc.health.smithi088.stdout: 2023-10-18T16:07:27.343 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"fb1f77e8-ead0-4f90-8128-fdf5137eb16f","created":"2023-10-18T16:07:23.662468+0000","modified":"2023-10-18T16:07:26.064784+0000","last_up_change":"0.000000","last_in_change":"2023-10-18T16:07:26.064784+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"e421c20c-f7bc-4574-97a2-201bb4a16b4b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9485c5ec-379a-4038-a2d8-1f1081fe2eea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"671895e2-57f3-4f1a-ba4a-a580af1f3ec2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0c4e6e7f-7d9c-4e65-beec-fd0ca384f320","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"29570691-ef35-4b80-9247-aa3703877af4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"26540b7b-032a-4b3b-bf6c-b131a449fa99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7af01f9c-90db-45fd-93c8-09517c9b0881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b527b0a5-4249-4854-a30f-bff897757239","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-18T16:07:27.344 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.340+0000 7f0bba62a700 1 -- 172.21.15.88:0/1711963976 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0bb4064390 msgr2=0x7f0bb40b7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:27.344 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.340+0000 7f0bba62a700 1 --2- 172.21.15.88:0/1711963976 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0bb4064390 0x7f0bb40b7230 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f0bb0001eb0 tx=0x7f0bb0007e90).stop 2023-10-18T16:07:27.345 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.340+0000 7f0bba62a700 1 -- 172.21.15.88:0/1711963976 shutdown_connections 2023-10-18T16:07:27.345 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.340+0000 7f0bba62a700 1 --2- 172.21.15.88:0/1711963976 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bb40bfa10 0x7f0bb4161d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:27.345 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.340+0000 7f0bba62a700 1 --2- 172.21.15.88:0/1711963976 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0bb4064390 0x7f0bb40b7230 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:27.345 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.340+0000 7f0bba62a700 1 --2- 172.21.15.88:0/1711963976 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0bb4066be0 0x7f0bb414dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:27.345 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.340+0000 7f0bba62a700 1 -- 172.21.15.88:0/1711963976 >> 172.21.15.88:0/1711963976 conn(0x7f0bb40c3f60 msgr2=0x7f0bb4152f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:27.346 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.340+0000 7f0bba62a700 1 -- 172.21.15.88:0/1711963976 shutdown_connections 2023-10-18T16:07:27.346 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:27.340+0000 7f0bba62a700 1 -- 172.21.15.88:0/1711963976 wait complete. 2023-10-18T16:07:27.355 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-10-18T16:07:27.460 INFO:tasks.ceph.mgr.x.smithi151.stderr:2023-10-18T16:07:27.456+0000 7f8d9db54680 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-18T16:07:27.498 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-10-18T16:07:27.495+0000 7f07f6041680 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-18T16:07:27.528 INFO:tasks.ceph.mgr.x.smithi151.stderr:2023-10-18T16:07:27.524+0000 7f8d9db54680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-18T16:07:27.570 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-10-18T16:07:27.567+0000 7f07f6041680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-18T16:07:27.606 INFO:tasks.ceph.mgr.x.smithi151.stderr:2023-10-18T16:07:27.603+0000 7f8d9db54680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-18T16:07:27.650 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-10-18T16:07:27.647+0000 7f07f6041680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-18T16:07:27.752 INFO:tasks.ceph.mgr.x.smithi151.stderr:2023-10-18T16:07:27.749+0000 7f8d9db54680 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-18T16:07:27.803 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-10-18T16:07:27.800+0000 7f07f6041680 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-18T16:07:27.824 INFO:tasks.ceph.mgr.x.smithi151.stderr:2023-10-18T16:07:27.820+0000 7f8d9db54680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-18T16:07:27.876 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-10-18T16:07:27.873+0000 7f07f6041680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-18T16:07:27.939 INFO:tasks.ceph.mgr.x.smithi151.stderr:2023-10-18T16:07:27.936+0000 7f8d9db54680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-18T16:07:27.996 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-10-18T16:07:27.993+0000 7f07f6041680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-18T16:07:28.045 INFO:tasks.ceph.mgr.x.smithi151.stderr:2023-10-18T16:07:28.041+0000 7f8d9db54680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-18T16:07:28.107 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-10-18T16:07:28.104+0000 7f07f6041680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-18T16:07:28.366 INFO:tasks.ceph.mgr.x.smithi151.stderr:2023-10-18T16:07:28.363+0000 7f8d9db54680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-18T16:07:28.435 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-10-18T16:07:28.432+0000 7f07f6041680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-18T16:07:28.437 INFO:tasks.ceph.mgr.x.smithi151.stderr:2023-10-18T16:07:28.433+0000 7f8d9db54680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-18T16:07:28.511 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-10-18T16:07:28.508+0000 7f07f6041680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-18T16:07:30.177 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-10-18T16:07:30.174+0000 7f955ca9c200 -1 osd.0 0 log_to_monitors {default=true} 2023-10-18T16:07:30.184 INFO:tasks.ceph.osd.4.smithi151.stderr:2023-10-18T16:07:30.180+0000 7f1d41306200 -1 osd.4 0 log_to_monitors {default=true} 2023-10-18T16:07:30.209 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-10-18T16:07:30.206+0000 7f0a8dbe1200 -1 osd.2 0 log_to_monitors {default=true} 2023-10-18T16:07:30.235 INFO:tasks.ceph.osd.5.smithi151.stderr:2023-10-18T16:07:30.231+0000 7f383af81200 -1 osd.5 0 log_to_monitors {default=true} 2023-10-18T16:07:30.249 INFO:tasks.ceph.osd.7.smithi151.stderr:2023-10-18T16:07:30.246+0000 7f1205428200 -1 osd.7 0 log_to_monitors {default=true} 2023-10-18T16:07:30.256 INFO:tasks.ceph.osd.6.smithi151.stderr:2023-10-18T16:07:30.252+0000 7f8145eb5200 -1 osd.6 0 log_to_monitors {default=true} 2023-10-18T16:07:30.465 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-10-18T16:07:30.462+0000 7fd71bb78200 -1 osd.3 0 log_to_monitors {default=true} 2023-10-18T16:07:30.475 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-10-18T16:07:30.471+0000 7f25e92d3200 -1 osd.1 0 log_to_monitors {default=true} 2023-10-18T16:07:30.504 INFO:tasks.ceph.mgr.x.smithi151.stderr:2023-10-18T16:07:30.500+0000 7f8d53d77700 -1 mgr.server handle_report got status from non-daemon mon.b 2023-10-18T16:07:31.498 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-10-18T16:07:31.494+0000 7fd712536700 -1 osd.3 0 waiting for initial osdmap 2023-10-18T16:07:31.499 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-10-18T16:07:31.495+0000 7f955345a700 -1 osd.0 0 waiting for initial osdmap 2023-10-18T16:07:31.500 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-10-18T16:07:31.495+0000 7f25dfc91700 -1 osd.1 0 waiting for initial osdmap 2023-10-18T16:07:31.501 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-10-18T16:07:31.496+0000 7f0a8459f700 -1 osd.2 0 waiting for initial osdmap 2023-10-18T16:07:31.501 INFO:tasks.ceph.osd.4.smithi151.stderr:2023-10-18T16:07:31.496+0000 7f1d37cc4700 -1 osd.4 0 waiting for initial osdmap 2023-10-18T16:07:31.502 INFO:tasks.ceph.osd.7.smithi151.stderr:2023-10-18T16:07:31.497+0000 7f11fbde6700 -1 osd.7 0 waiting for initial osdmap 2023-10-18T16:07:31.503 INFO:tasks.ceph.osd.5.smithi151.stderr:2023-10-18T16:07:31.497+0000 7f383193f700 -1 osd.5 0 waiting for initial osdmap 2023-10-18T16:07:31.503 INFO:tasks.ceph.osd.6.smithi151.stderr:2023-10-18T16:07:31.498+0000 7f813c873700 -1 osd.6 0 waiting for initial osdmap 2023-10-18T16:07:31.504 INFO:tasks.ceph.mgr.x.smithi151.stderr:2023-10-18T16:07:31.499+0000 7f8d53d77700 -1 mgr.server handle_report got status from non-daemon mon.c 2023-10-18T16:07:31.505 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-10-18T16:07:31.500+0000 7f25dc40d700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-18T16:07:31.506 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-10-18T16:07:31.499+0000 7fd70ecb2700 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-18T16:07:31.506 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-10-18T16:07:31.501+0000 7f0a80d1b700 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-18T16:07:31.507 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-10-18T16:07:31.501+0000 7f954fbd6700 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-18T16:07:31.511 INFO:tasks.ceph.osd.6.smithi151.stderr:2023-10-18T16:07:31.508+0000 7f8138fef700 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-18T16:07:31.512 INFO:tasks.ceph.osd.7.smithi151.stderr:2023-10-18T16:07:31.508+0000 7f11f8562700 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-18T16:07:31.514 INFO:tasks.ceph.osd.4.smithi151.stderr:2023-10-18T16:07:31.511+0000 7f1d34440700 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-18T16:07:31.514 INFO:tasks.ceph.osd.5.smithi151.stderr:2023-10-18T16:07:31.511+0000 7f382e0bb700 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-18T16:07:34.162 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-18T16:07:34.266 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.262+0000 7fe37f59e700 1 Processor -- start 2023-10-18T16:07:34.266 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.262+0000 7fe37f59e700 1 -- start start 2023-10-18T16:07:34.269 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.263+0000 7fe37f59e700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fe3800698a0 0x7fe380150890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:34.270 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.263+0000 7fe37f59e700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3800681a0 con 0x7fe380069c80 2023-10-18T16:07:34.270 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.263+0000 7fe37f59e700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3800bd950 con 0x7fe380068db0 2023-10-18T16:07:34.270 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.263+0000 7fe37f59e700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3800bf960 con 0x7fe3800698a0 2023-10-18T16:07:34.270 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.263+0000 7fe37dd9b700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fe380069c80 0x7fe3800686a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:50824/0 (socket says 172.21.15.88:50824) 2023-10-18T16:07:34.270 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.263+0000 7fe37dd9b700 1 -- 172.21.15.88:0/950339624 learned_addr learned my addr 172.21.15.88:0/950339624 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:07:34.271 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.263+0000 7fe37e59c700 1 --2- 172.21.15.88:0/950339624 >> v2:172.21.15.151:3300/0 conn(0x7fe3800698a0 0x7fe380150890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:34.271 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.263+0000 7fe37d59a700 1 -- 172.21.15.88:0/950339624 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1347340833 0 0) 0x7fe3800681a0 con 0x7fe380069c80 2023-10-18T16:07:34.271 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.263+0000 7fe37d59a700 1 -- 172.21.15.88:0/950339624 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3680033f0 con 0x7fe380069c80 2023-10-18T16:07:34.271 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.263+0000 7fe37d59a700 1 -- 172.21.15.88:0/950339624 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 149122915 0 0) 0x7fe3680033f0 con 0x7fe380069c80 2023-10-18T16:07:34.271 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.263+0000 7fe37d59a700 1 -- 172.21.15.88:0/950339624 >> v2:172.21.15.151:3300/0 conn(0x7fe3800698a0 msgr2=0x7fe380150890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:34.272 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.263+0000 7fe37d59a700 1 --2- 172.21.15.88:0/950339624 >> v2:172.21.15.151:3300/0 conn(0x7fe3800698a0 0x7fe380150890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:34.272 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.263+0000 7fe37d59a700 1 -- 172.21.15.88:0/950339624 >> v1:172.21.15.151:6789/0 conn(0x7fe380068db0 legacy=0x7fe3800bff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:34.272 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.263+0000 7fe37d59a700 1 -- 172.21.15.88:0/950339624 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3800bdc30 con 0x7fe380069c80 2023-10-18T16:07:34.272 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.263+0000 7fe37d59a700 1 -- 172.21.15.88:0/950339624 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441715075 0 0) 0x7fe374001470 con 0x7fe380069c80 2023-10-18T16:07:34.272 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.264+0000 7fe37d59a700 1 -- 172.21.15.88:0/950339624 >> v1:172.21.15.88:6789/0 conn(0x7fe380069c80 legacy=0x7fe3800686a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:34.273 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.264+0000 7fe37d59a700 1 --2- 172.21.15.88:0/950339624 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3680038e0 0x7fe368003d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:34.273 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.264+0000 7fe37d59a700 1 --2- 172.21.15.88:0/950339624 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe368004690 0x7fe368004b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:34.273 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.264+0000 7fe37d59a700 1 --2- 172.21.15.88:0/950339624 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe368005000 0x7fe36800b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:34.273 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.264+0000 7fe37d59a700 1 -- 172.21.15.88:0/950339624 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3800bf960 con 0x7fe3680038e0 2023-10-18T16:07:34.273 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.264+0000 7fe37d59a700 1 -- 172.21.15.88:0/950339624 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe36800beb0 con 0x7fe368004690 2023-10-18T16:07:34.274 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.264+0000 7fe37d59a700 1 -- 172.21.15.88:0/950339624 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe36800c070 con 0x7fe368005000 2023-10-18T16:07:34.274 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.264+0000 7fe37e59c700 1 --2- 172.21.15.88:0/950339624 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3680038e0 0x7fe368003d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:34.274 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.264+0000 7fe37dd9b700 1 --2- 172.21.15.88:0/950339624 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe368004690 0x7fe368004b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:34.274 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.264+0000 7fe37ed9d700 1 --2- 172.21.15.88:0/950339624 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe368005000 0x7fe36800b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:34.274 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.264+0000 7fe37e59c700 1 -- 172.21.15.88:0/950339624 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe368005000 msgr2=0x7fe36800b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:34.275 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.264+0000 7fe37e59c700 1 --2- 172.21.15.88:0/950339624 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe368005000 0x7fe36800b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:34.275 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.264+0000 7fe37e59c700 1 -- 172.21.15.88:0/950339624 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe368004690 msgr2=0x7fe368004b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:34.275 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.264+0000 7fe37e59c700 1 --2- 172.21.15.88:0/950339624 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe368004690 0x7fe368004b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:34.275 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.264+0000 7fe37e59c700 1 -- 172.21.15.88:0/950339624 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3680033f0 con 0x7fe3680038e0 2023-10-18T16:07:34.275 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.264+0000 7fe37ed9d700 1 -- 172.21.15.88:0/950339624 reap_dead start 2023-10-18T16:07:34.276 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.264+0000 7fe37e59c700 1 --2- 172.21.15.88:0/950339624 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3680038e0 0x7fe368003d00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fe378009a80 tx=0x7fe378009060).ready entity=mon.0 client_cookie=a5dbda5072470417 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:34.276 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.264+0000 7fe37d59a700 1 -- 172.21.15.88:0/950339624 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe378009430 con 0x7fe3680038e0 2023-10-18T16:07:34.276 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.264+0000 7fe37d59a700 1 -- 172.21.15.88:0/950339624 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe37800e640 con 0x7fe3680038e0 2023-10-18T16:07:34.276 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.264+0000 7fe37d59a700 1 -- 172.21.15.88:0/950339624 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe37800e9c0 con 0x7fe3680038e0 2023-10-18T16:07:34.276 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.265+0000 7fe37f59e700 1 -- 172.21.15.88:0/950339624 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3680038e0 msgr2=0x7fe368003d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:34.277 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.265+0000 7fe37f59e700 1 --2- 172.21.15.88:0/950339624 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3680038e0 0x7fe368003d00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fe378009a80 tx=0x7fe378009060).stop 2023-10-18T16:07:34.277 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.265+0000 7fe37f59e700 1 -- 172.21.15.88:0/950339624 shutdown_connections 2023-10-18T16:07:34.277 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.265+0000 7fe37f59e700 1 --2- 172.21.15.88:0/950339624 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3680038e0 0x7fe368003d00 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:34.277 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.265+0000 7fe37f59e700 1 -- 172.21.15.88:0/950339624 >> 172.21.15.88:0/950339624 conn(0x7fe3800c6770 msgr2=0x7fe3800c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:34.277 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.265+0000 7fe37f59e700 1 -- 172.21.15.88:0/950339624 shutdown_connections 2023-10-18T16:07:34.278 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.265+0000 7fe37f59e700 1 -- 172.21.15.88:0/950339624 wait complete. 2023-10-18T16:07:34.278 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.266+0000 7fe37f59e700 1 Processor -- start 2023-10-18T16:07:34.278 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.266+0000 7fe37f59e700 1 -- start start 2023-10-18T16:07:34.280 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.266+0000 7fe37f59e700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe3680038e0 0x7fe380154830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:34.281 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.266+0000 7fe37f59e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe380154180 0x7fe38014acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:34.281 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.267+0000 7fe37f59e700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe380154d60 0x7fe3800c5570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:34.281 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.267+0000 7fe37f59e700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe380154670 con 0x7fe380154d60 2023-10-18T16:07:34.281 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.267+0000 7fe37f59e700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe380163950 con 0x7fe380154180 2023-10-18T16:07:34.282 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.267+0000 7fe37f59e700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe380152bd0 con 0x7fe3680038e0 2023-10-18T16:07:34.282 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.267+0000 7fe37ed9d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe380154d60 0x7fe3800c5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:34.282 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.267+0000 7fe37ed9d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe380154d60 0x7fe3800c5570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:34520/0 (socket says 172.21.15.88:34520) 2023-10-18T16:07:34.282 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.267+0000 7fe37ed9d700 1 -- 172.21.15.88:0/4048428039 learned_addr learned my addr 172.21.15.88:0/4048428039 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:34.282 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.267+0000 7fe37e59c700 1 --2- 172.21.15.88:0/4048428039 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe3680038e0 0x7fe380154830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:34.283 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.267+0000 7fe37dd9b700 1 --2- 172.21.15.88:0/4048428039 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe380154180 0x7fe38014acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:34.283 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.267+0000 7fe37ed9d700 1 -- 172.21.15.88:0/4048428039 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe3680038e0 msgr2=0x7fe380154830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:34.283 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.267+0000 7fe37ed9d700 1 --2- 172.21.15.88:0/4048428039 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe3680038e0 0x7fe380154830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:34.283 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.267+0000 7fe37ed9d700 1 -- 172.21.15.88:0/4048428039 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe380154180 msgr2=0x7fe38014acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:34.283 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.267+0000 7fe37ed9d700 1 --2- 172.21.15.88:0/4048428039 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe380154180 0x7fe38014acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:34.284 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.267+0000 7fe37ed9d700 1 -- 172.21.15.88:0/4048428039 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe380152eb0 con 0x7fe380154d60 2023-10-18T16:07:34.284 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.267+0000 7fe37dd9b700 1 --2- 172.21.15.88:0/4048428039 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe380154180 0x7fe38014acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:07:34.284 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.267+0000 7fe37ed9d700 1 --2- 172.21.15.88:0/4048428039 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe380154d60 0x7fe3800c5570 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fe380084370 tx=0x7fe370009040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:34.284 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.267+0000 7fe36e7fc700 1 -- 172.21.15.88:0/4048428039 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe37000eea0 con 0x7fe380154d60 2023-10-18T16:07:34.284 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.267+0000 7fe36e7fc700 1 -- 172.21.15.88:0/4048428039 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3700030d0 con 0x7fe380154d60 2023-10-18T16:07:34.285 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.267+0000 7fe37f59e700 1 -- 172.21.15.88:0/4048428039 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe380153190 con 0x7fe380154d60 2023-10-18T16:07:34.285 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.267+0000 7fe36e7fc700 1 -- 172.21.15.88:0/4048428039 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe370001780 con 0x7fe380154d60 2023-10-18T16:07:34.285 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.267+0000 7fe37f59e700 1 -- 172.21.15.88:0/4048428039 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe38014b1b0 con 0x7fe380154d60 2023-10-18T16:07:34.285 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.268+0000 7fe37f59e700 1 -- 172.21.15.88:0/4048428039 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe380153630 con 0x7fe380154d60 2023-10-18T16:07:34.285 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.268+0000 7fe36e7fc700 1 -- 172.21.15.88:0/4048428039 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7fe370003450 con 0x7fe380154d60 2023-10-18T16:07:34.286 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.269+0000 7fe36e7fc700 1 --2- 172.21.15.88:0/4048428039 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fe36005f180 0x7fe360061640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:34.286 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.269+0000 7fe36e7fc700 1 -- 172.21.15.88:0/4048428039 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 4892+0+0 (secure 0 0 0) 0x7fe370079160 con 0x7fe380154d60 2023-10-18T16:07:34.286 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.269+0000 7fe37e59c700 1 --2- 172.21.15.88:0/4048428039 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fe36005f180 0x7fe360061640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:34.286 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.269+0000 7fe37e59c700 1 --2- 172.21.15.88:0/4048428039 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fe36005f180 0x7fe360061640 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fe378006d50 tx=0x7fe37801a000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:34.286 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.273+0000 7fe36e7fc700 1 -- 172.21.15.88:0/4048428039 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7fe37004a900 con 0x7fe380154d60 2023-10-18T16:07:34.446 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.443+0000 7fe37f59e700 1 -- 172.21.15.88:0/4048428039 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe3800c5e80 con 0x7fe380154d60 2023-10-18T16:07:34.447 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.444+0000 7fe36e7fc700 1 -- 172.21.15.88:0/4048428039 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) v1 ==== 74+0+12180 (secure 0 0 0) 0x7fe37001c090 con 0x7fe380154d60 2023-10-18T16:07:34.448 INFO:teuthology.misc.health.smithi088.stdout: 2023-10-18T16:07:34.448 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":14,"fsid":"fb1f77e8-ead0-4f90-8128-fdf5137eb16f","created":"2023-10-18T16:07:23.662468+0000","modified":"2023-10-18T16:07:33.506699+0000","last_up_change":"2023-10-18T16:07:32.498203+0000","last_in_change":"2023-10-18T16:07:26.064784+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-10-18T16:07:32.536320+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"e421c20c-f7bc-4574-97a2-201bb4a16b4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":91007},{"type":"v1","addr":"172.21.15.88:6809","nonce":91007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":91007},{"type":"v1","addr":"172.21.15.88:6811","nonce":91007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":91007},{"type":"v1","addr":"172.21.15.88:6815","nonce":91007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":91007},{"type":"v1","addr":"172.21.15.88:6813","nonce":91007}]},"public_addr":"172.21.15.88:6809/91007","cluster_addr":"172.21.15.88:6811/91007","heartbeat_back_addr":"172.21.15.88:6815/91007","heartbeat_front_addr":"172.21.15.88:6813/91007","state":["exists","up"]},{"osd":1,"uuid":"9485c5ec-379a-4038-a2d8-1f1081fe2eea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":91009},{"type":"v1","addr":"172.21.15.88:6825","nonce":91009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":91009},{"type":"v1","addr":"172.21.15.88:6827","nonce":91009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":91009},{"type":"v1","addr":"172.21.15.88:6831","nonce":91009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":91009},{"type":"v1","addr":"172.21.15.88:6829","nonce":91009}]},"public_addr":"172.21.15.88:6825/91009","cluster_addr":"172.21.15.88:6827/91009","heartbeat_back_addr":"172.21.15.88:6831/91009","heartbeat_front_addr":"172.21.15.88:6829/91009","state":["exists","up"]},{"osd":2,"uuid":"671895e2-57f3-4f1a-ba4a-a580af1f3ec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":91005},{"type":"v1","addr":"172.21.15.88:6801","nonce":91005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":91005},{"type":"v1","addr":"172.21.15.88:6803","nonce":91005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":91005},{"type":"v1","addr":"172.21.15.88:6807","nonce":91005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":91005},{"type":"v1","addr":"172.21.15.88:6805","nonce":91005}]},"public_addr":"172.21.15.88:6801/91005","cluster_addr":"172.21.15.88:6803/91005","heartbeat_back_addr":"172.21.15.88:6807/91005","heartbeat_front_addr":"172.21.15.88:6805/91005","state":["exists","up"]},{"osd":3,"uuid":"0c4e6e7f-7d9c-4e65-beec-fd0ca384f320","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":91008},{"type":"v1","addr":"172.21.15.88:6817","nonce":91008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":91008},{"type":"v1","addr":"172.21.15.88:6819","nonce":91008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":91008},{"type":"v1","addr":"172.21.15.88:6823","nonce":91008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":91008},{"type":"v1","addr":"172.21.15.88:6821","nonce":91008}]},"public_addr":"172.21.15.88:6817/91008","cluster_addr":"172.21.15.88:6819/91008","heartbeat_back_addr":"172.21.15.88:6823/91008","heartbeat_front_addr":"172.21.15.88:6821/91008","state":["exists","up"]},{"osd":4,"uuid":"29570691-ef35-4b80-9247-aa3703877af4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":91514},{"type":"v1","addr":"172.21.15.151:6801","nonce":91514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":91514},{"type":"v1","addr":"172.21.15.151:6803","nonce":91514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":91514},{"type":"v1","addr":"172.21.15.151:6807","nonce":91514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":91514},{"type":"v1","addr":"172.21.15.151:6805","nonce":91514}]},"public_addr":"172.21.15.151:6801/91514","cluster_addr":"172.21.15.151:6803/91514","heartbeat_back_addr":"172.21.15.151:6807/91514","heartbeat_front_addr":"172.21.15.151:6805/91514","state":["exists","up"]},{"osd":5,"uuid":"26540b7b-032a-4b3b-bf6c-b131a449fa99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":91513},{"type":"v1","addr":"172.21.15.151:6809","nonce":91513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":91513},{"type":"v1","addr":"172.21.15.151:6811","nonce":91513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":91513},{"type":"v1","addr":"172.21.15.151:6815","nonce":91513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":91513},{"type":"v1","addr":"172.21.15.151:6813","nonce":91513}]},"public_addr":"172.21.15.151:6809/91513","cluster_addr":"172.21.15.151:6811/91513","heartbeat_back_addr":"172.21.15.151:6815/91513","heartbeat_front_addr":"172.21.15.151:6813/91513","state":["exists","up"]},{"osd":6,"uuid":"7af01f9c-90db-45fd-93c8-09517c9b0881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":91544},{"type":"v1","addr":"172.21.15.151:6817","nonce":91544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":91544},{"type":"v1","addr":"172.21.15.151:6819","nonce":91544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":91544},{"type":"v1","addr":"172.21.15.151:6823","nonce":91544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":91544},{"type":"v1","addr":"172.21.15.151:6821","nonce":91544}]},"public_addr":"172.21.15.151:6817/91544","cluster_addr":"172.21.15.151:6819/91544","heartbeat_back_addr":"172.21.15.151:6823/91544","heartbeat_front_addr":"172.21.15.151:6821/91544","state":["exists","up"]},{"osd":7,"uuid":"b527b0a5-4249-4854-a30f-bff897757239","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":91545},{"type":"v1","addr":"172.21.15.151:6825","nonce":91545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":91545},{"type":"v1","addr":"172.21.15.151:6827","nonce":91545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":91545},{"type":"v1","addr":"172.21.15.151:6831","nonce":91545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":91545},{"type":"v1","addr":"172.21.15.151:6829","nonce":91545}]},"public_addr":"172.21.15.151:6825/91545","cluster_addr":"172.21.15.151:6827/91545","heartbeat_back_addr":"172.21.15.151:6831/91545","heartbeat_front_addr":"172.21.15.151:6829/91545","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T16:07:31.216742+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T16:07:31.467229+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T16:07:31.220809+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T16:07:31.414999+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T16:07:31.188976+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T16:07:31.252153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T16:07:31.254324+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T16:07:31.203434+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-18T16:07:34.449 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.446+0000 7fe37f59e700 1 -- 172.21.15.88:0/4048428039 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fe36005f180 msgr2=0x7fe360061640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:34.450 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.446+0000 7fe37f59e700 1 --2- 172.21.15.88:0/4048428039 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fe36005f180 0x7fe360061640 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fe378006d50 tx=0x7fe37801a000).stop 2023-10-18T16:07:34.450 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.446+0000 7fe37f59e700 1 -- 172.21.15.88:0/4048428039 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe380154d60 msgr2=0x7fe3800c5570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:34.450 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.446+0000 7fe37f59e700 1 --2- 172.21.15.88:0/4048428039 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe380154d60 0x7fe3800c5570 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fe380084370 tx=0x7fe370009040).stop 2023-10-18T16:07:34.450 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.446+0000 7fe37f59e700 1 -- 172.21.15.88:0/4048428039 shutdown_connections 2023-10-18T16:07:34.450 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.446+0000 7fe37f59e700 1 --2- 172.21.15.88:0/4048428039 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fe36005f180 0x7fe360061640 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:34.451 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.446+0000 7fe37f59e700 1 --2- 172.21.15.88:0/4048428039 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe380154d60 0x7fe3800c5570 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:34.451 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.446+0000 7fe37f59e700 1 --2- 172.21.15.88:0/4048428039 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe3680038e0 0x7fe380154830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:34.451 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.446+0000 7fe37f59e700 1 --2- 172.21.15.88:0/4048428039 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe380154180 0x7fe38014acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:34.451 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.446+0000 7fe37f59e700 1 -- 172.21.15.88:0/4048428039 >> 172.21.15.88:0/4048428039 conn(0x7fe3800c6770 msgr2=0x7fe38014e160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:34.452 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.446+0000 7fe37f59e700 1 -- 172.21.15.88:0/4048428039 shutdown_connections 2023-10-18T16:07:34.452 INFO:teuthology.misc.health.smithi088.stderr:2023-10-18T16:07:34.446+0000 7fe37f59e700 1 -- 172.21.15.88:0/4048428039 wait complete. 2023-10-18T16:07:34.460 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-10-18T16:07:34.461 INFO:tasks.ceph:Creating RBD pool 2023-10-18T16:07:34.461 DEBUG:teuthology.orchestra.run.smithi088:> sudo ceph --cluster ceph osd pool create rbd 8 2023-10-18T16:07:34.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:34.575+0000 7ff51c8ae700 1 Processor -- start 2023-10-18T16:07:34.579 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:34.575+0000 7ff51c8ae700 1 -- start start 2023-10-18T16:07:34.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:34.576+0000 7ff51c8ae700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7ff5180bd7e0 0x7ff51814e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:34.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:34.576+0000 7ff51c8ae700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff5180a8c20 0x7ff5180bdbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:34.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:34.576+0000 7ff51c8ae700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5180bfe30 con 0x7ff5180be0b0 2023-10-18T16:07:34.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:34.576+0000 7ff51c8ae700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff51814edc0 con 0x7ff5180a8c20 2023-10-18T16:07:34.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:34.576+0000 7ff51c8ae700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5180c03f0 con 0x7ff5180bd7e0 2023-10-18T16:07:34.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:34.576+0000 7ff517fff700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7ff5180be0b0 0x7ff518069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:50840/0 (socket says 172.21.15.88:50840) 2023-10-18T16:07:34.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:34.576+0000 7ff516ffd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff5180a8c20 0x7ff5180bdbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:34.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:34.576+0000 7ff517fff700 1 -- 172.21.15.88:0/347765736 learned_addr learned my addr 172.21.15.88:0/347765736 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:07:34.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:34.576+0000 7ff5177fe700 1 --2- 172.21.15.88:0/347765736 >> v2:172.21.15.151:3300/0 conn(0x7ff5180bd7e0 0x7ff51814e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:34.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:34.576+0000 7ff516ffd700 1 -- 172.21.15.88:0/347765736 >> v2:172.21.15.151:3300/0 conn(0x7ff5180bd7e0 msgr2=0x7ff51814e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:34.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:34.576+0000 7ff516ffd700 1 --2- 172.21.15.88:0/347765736 >> v2:172.21.15.151:3300/0 conn(0x7ff5180bd7e0 0x7ff51814e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:34.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:34.576+0000 7ff516ffd700 1 -- 172.21.15.88:0/347765736 >> v1:172.21.15.88:6789/0 conn(0x7ff5180be0b0 legacy=0x7ff518069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:34.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:34.576+0000 7ff516ffd700 1 -- 172.21.15.88:0/347765736 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff518069110 con 0x7ff5180a8c20 2023-10-18T16:07:34.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:34.576+0000 7ff5167fc700 1 -- 172.21.15.88:0/347765736 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 121086996 0 0) 0x7ff5180bfe30 con 0x7ff5180be0b0 2023-10-18T16:07:34.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:34.576+0000 7ff516ffd700 1 --2- 172.21.15.88:0/347765736 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5180a8c20 0x7ff5180bdbc0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7ff5180665e0 tx=0x7ff50c009060).ready entity=mon.0 client_cookie=cfa3d3dddac93da6 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:34.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:34.576+0000 7ff5167fc700 1 -- 172.21.15.88:0/347765736 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff50c00f510 con 0x7ff5180a8c20 2023-10-18T16:07:34.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:34.576+0000 7ff5167fc700 1 -- 172.21.15.88:0/347765736 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff50c00f670 con 0x7ff5180a8c20 2023-10-18T16:07:34.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:34.576+0000 7ff5167fc700 1 -- 172.21.15.88:0/347765736 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff50c00f9f0 con 0x7ff5180a8c20 2023-10-18T16:07:34.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:34.577+0000 7ff51c8ae700 1 -- 172.21.15.88:0/347765736 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5180a8c20 msgr2=0x7ff5180bdbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:34.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:34.577+0000 7ff51c8ae700 1 --2- 172.21.15.88:0/347765736 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5180a8c20 0x7ff5180bdbc0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7ff5180665e0 tx=0x7ff50c009060).stop 2023-10-18T16:07:34.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:34.577+0000 7ff51c8ae700 1 -- 172.21.15.88:0/347765736 shutdown_connections 2023-10-18T16:07:34.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:34.577+0000 7ff51c8ae700 1 --2- 172.21.15.88:0/347765736 >> v2:172.21.15.151:3300/0 conn(0x7ff5180bd7e0 0x7ff51814e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:34.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:34.577+0000 7ff51c8ae700 1 --2- 172.21.15.88:0/347765736 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5180a8c20 0x7ff5180bdbc0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:34.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:34.577+0000 7ff51c8ae700 1 -- 172.21.15.88:0/347765736 >> 172.21.15.88:0/347765736 conn(0x7ff5180c3750 msgr2=0x7ff5180bf720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:34.586 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:34.577+0000 7ff51c8ae700 1 -- 172.21.15.88:0/347765736 shutdown_connections 2023-10-18T16:07:34.586 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:34.577+0000 7ff51c8ae700 1 -- 172.21.15.88:0/347765736 wait complete. 2023-10-18T16:07:34.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:34.578+0000 7ff51c8ae700 1 Processor -- start 2023-10-18T16:07:34.589 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:34.578+0000 7ff51c8ae700 1 -- start start 2023-10-18T16:07:34.589 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:34.579+0000 7ff51c8ae700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5180a8c20 0x7ff51814aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:34.589 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:34.579+0000 7ff51c8ae700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff5180bd7e0 0x7ff518151950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:34.589 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:34.579+0000 7ff51c8ae700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff5180be0b0 0x7ff518151e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:34.589 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:34.579+0000 7ff51c8ae700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff51814d910 con 0x7ff5180a8c20 2023-10-18T16:07:34.590 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:34.579+0000 7ff51c8ae700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff518164400 con 0x7ff5180be0b0 2023-10-18T16:07:34.590 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:34.579+0000 7ff51c8ae700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff518163f10 con 0x7ff5180bd7e0 2023-10-18T16:07:34.590 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:34.579+0000 7ff5177fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5180a8c20 0x7ff51814aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:34.590 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:34.579+0000 7ff5177fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5180a8c20 0x7ff51814aac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:34536/0 (socket says 172.21.15.88:34536) 2023-10-18T16:07:34.591 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:34.579+0000 7ff5177fe700 1 -- 172.21.15.88:0/1523501375 learned_addr learned my addr 172.21.15.88:0/1523501375 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:34.591 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:34.579+0000 7ff517fff700 1 --2- 172.21.15.88:0/1523501375 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff5180be0b0 0x7ff518151e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:34.591 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:34.579+0000 7ff516ffd700 1 --2- 172.21.15.88:0/1523501375 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff5180bd7e0 0x7ff518151950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:34.591 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:34.579+0000 7ff5177fe700 1 -- 172.21.15.88:0/1523501375 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff5180bd7e0 msgr2=0x7ff518151950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:34.591 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:34.579+0000 7ff5177fe700 1 --2- 172.21.15.88:0/1523501375 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff5180bd7e0 0x7ff518151950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:34.591 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:34.579+0000 7ff5177fe700 1 -- 172.21.15.88:0/1523501375 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff5180be0b0 msgr2=0x7ff518151e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:34.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:34.579+0000 7ff5177fe700 1 --2- 172.21.15.88:0/1523501375 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff5180be0b0 0x7ff518151e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:34.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:34.579+0000 7ff5177fe700 1 -- 172.21.15.88:0/1523501375 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff50c0092a0 con 0x7ff5180a8c20 2023-10-18T16:07:34.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:34.579+0000 7ff5177fe700 1 --2- 172.21.15.88:0/1523501375 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5180a8c20 0x7ff51814aac0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7ff50800e990 tx=0x7ff508009060).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:34.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:34.579+0000 7ff4ff7fe700 1 -- 172.21.15.88:0/1523501375 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff508009510 con 0x7ff5180a8c20 2023-10-18T16:07:34.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:34.579+0000 7ff4ff7fe700 1 -- 172.21.15.88:0/1523501375 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff50800c670 con 0x7ff5180a8c20 2023-10-18T16:07:34.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:34.579+0000 7ff4ff7fe700 1 -- 172.21.15.88:0/1523501375 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff50800c9f0 con 0x7ff5180a8c20 2023-10-18T16:07:34.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:34.579+0000 7ff51c8ae700 1 -- 172.21.15.88:0/1523501375 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff51814d020 con 0x7ff5180a8c20 2023-10-18T16:07:34.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:34.580+0000 7ff51c8ae700 1 -- 172.21.15.88:0/1523501375 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5180ba4e0 con 0x7ff5180a8c20 2023-10-18T16:07:34.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:34.580+0000 7ff51c8ae700 1 -- 172.21.15.88:0/1523501375 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff518063b90 con 0x7ff5180a8c20 2023-10-18T16:07:34.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:34.581+0000 7ff4ff7fe700 1 -- 172.21.15.88:0/1523501375 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7ff50801b400 con 0x7ff5180a8c20 2023-10-18T16:07:34.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:34.581+0000 7ff4ff7fe700 1 --2- 172.21.15.88:0/1523501375 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7ff50405f150 0x7ff504061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:34.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:34.581+0000 7ff4ff7fe700 1 -- 172.21.15.88:0/1523501375 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4916+0+0 (secure 0 0 0) 0x7ff50807a0b0 con 0x7ff5180a8c20 2023-10-18T16:07:34.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:34.581+0000 7ff516ffd700 1 --2- 172.21.15.88:0/1523501375 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7ff50405f150 0x7ff504061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:34.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:34.582+0000 7ff516ffd700 1 --2- 172.21.15.88:0/1523501375 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7ff50405f150 0x7ff504061610 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7ff5180bd020 tx=0x7ff50c017000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:34.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:34.585+0000 7ff4ff7fe700 1 -- 172.21.15.88:0/1523501375 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7ff50804b7d0 con 0x7ff5180a8c20 2023-10-18T16:07:34.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:34.747+0000 7ff51c8ae700 1 -- 172.21.15.88:0/1523501375 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7ff5180bf140 con 0x7ff5180a8c20 2023-10-18T16:07:35.534 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.530+0000 7ff4ff7fe700 1 -- 172.21.15.88:0/1523501375 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v16) v1 ==== 109+0+0 (secure 0 0 0) 0x7ff50804edf0 con 0x7ff5180a8c20 2023-10-18T16:07:35.536 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.533+0000 7ff51c8ae700 1 -- 172.21.15.88:0/1523501375 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7ff50405f150 msgr2=0x7ff504061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:35.537 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.533+0000 7ff51c8ae700 1 --2- 172.21.15.88:0/1523501375 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7ff50405f150 0x7ff504061610 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7ff5180bd020 tx=0x7ff50c017000).stop 2023-10-18T16:07:35.537 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.533+0000 7ff51c8ae700 1 -- 172.21.15.88:0/1523501375 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5180a8c20 msgr2=0x7ff51814aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:35.537 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.533+0000 7ff51c8ae700 1 --2- 172.21.15.88:0/1523501375 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5180a8c20 0x7ff51814aac0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7ff50800e990 tx=0x7ff508009060).stop 2023-10-18T16:07:35.537 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.533+0000 7ff51c8ae700 1 -- 172.21.15.88:0/1523501375 shutdown_connections 2023-10-18T16:07:35.538 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.533+0000 7ff51c8ae700 1 --2- 172.21.15.88:0/1523501375 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7ff50405f150 0x7ff504061610 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:35.538 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.533+0000 7ff51c8ae700 1 --2- 172.21.15.88:0/1523501375 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff5180be0b0 0x7ff518151e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:35.538 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.533+0000 7ff51c8ae700 1 --2- 172.21.15.88:0/1523501375 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5180a8c20 0x7ff51814aac0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:35.538 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.533+0000 7ff51c8ae700 1 --2- 172.21.15.88:0/1523501375 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff5180bd7e0 0x7ff518151950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:35.538 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.533+0000 7ff51c8ae700 1 -- 172.21.15.88:0/1523501375 >> 172.21.15.88:0/1523501375 conn(0x7ff5180c3750 msgr2=0x7ff51814e360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:35.538 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.533+0000 7ff51c8ae700 1 -- 172.21.15.88:0/1523501375 shutdown_connections 2023-10-18T16:07:35.539 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.533+0000 7ff51c8ae700 1 -- 172.21.15.88:0/1523501375 wait complete. 2023-10-18T16:07:35.539 INFO:teuthology.orchestra.run.smithi088.stderr:pool 'rbd' created 2023-10-18T16:07:35.549 DEBUG:teuthology.orchestra.run.smithi088:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-10-18T16:07:35.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.662+0000 7f750ff0f700 1 Processor -- start 2023-10-18T16:07:35.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.662+0000 7f750ff0f700 1 -- start start 2023-10-18T16:07:35.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.663+0000 7f750ff0f700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f75080bc730 0x7f7508067b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:35.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.663+0000 7f750ff0f700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75080684e0 con 0x7f7508150690 2023-10-18T16:07:35.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.663+0000 7f750ff0f700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f750806a860 con 0x7f75080692d0 2023-10-18T16:07:35.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.663+0000 7f750ff0f700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75080bb9c0 con 0x7f75080bc730 2023-10-18T16:07:35.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.663+0000 7f750dcab700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f7508150690 0x7f7508150a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:50844/0 (socket says 172.21.15.88:50844) 2023-10-18T16:07:35.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.663+0000 7f750dcab700 1 -- 172.21.15.88:0/2136594859 learned_addr learned my addr 172.21.15.88:0/2136594859 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:07:35.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.663+0000 7f750e4ac700 1 --2- 172.21.15.88:0/2136594859 >> v2:172.21.15.151:3300/0 conn(0x7f75080bc730 0x7f7508067b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:35.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.663+0000 7f750cca9700 1 -- 172.21.15.88:0/2136594859 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 806998553 0 0) 0x7f75080684e0 con 0x7f7508150690 2023-10-18T16:07:35.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.663+0000 7f750cca9700 1 -- 172.21.15.88:0/2136594859 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74f40033f0 con 0x7f7508150690 2023-10-18T16:07:35.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.664+0000 7f750cca9700 1 -- 172.21.15.88:0/2136594859 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3513666999 0 0) 0x7f74f40033f0 con 0x7f7508150690 2023-10-18T16:07:35.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.664+0000 7f750cca9700 1 -- 172.21.15.88:0/2136594859 >> v2:172.21.15.151:3300/0 conn(0x7f75080bc730 msgr2=0x7f7508067b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:35.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.664+0000 7f750cca9700 1 --2- 172.21.15.88:0/2136594859 >> v2:172.21.15.151:3300/0 conn(0x7f75080bc730 0x7f7508067b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:35.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.664+0000 7f750cca9700 1 -- 172.21.15.88:0/2136594859 >> v1:172.21.15.151:6789/0 conn(0x7f75080692d0 legacy=0x7f75080bc020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:35.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.664+0000 7f750cca9700 1 -- 172.21.15.88:0/2136594859 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7508069c50 con 0x7f7508150690 2023-10-18T16:07:35.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.664+0000 7f750cca9700 1 -- 172.21.15.88:0/2136594859 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441715075 0 0) 0x7f75040015f0 con 0x7f7508150690 2023-10-18T16:07:35.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.664+0000 7f750cca9700 1 -- 172.21.15.88:0/2136594859 >> v1:172.21.15.88:6789/0 conn(0x7f7508150690 legacy=0x7f7508150a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:35.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.664+0000 7f750cca9700 1 --2- 172.21.15.88:0/2136594859 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74f40038e0 0x7f74f4003d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:35.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.664+0000 7f750cca9700 1 --2- 172.21.15.88:0/2136594859 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f74f4004690 0x7f74f4004b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:35.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.664+0000 7f750cca9700 1 --2- 172.21.15.88:0/2136594859 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f74f4005000 0x7f74f400b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:35.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.664+0000 7f750cca9700 1 -- 172.21.15.88:0/2136594859 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75080bb9c0 con 0x7f74f40038e0 2023-10-18T16:07:35.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.664+0000 7f750cca9700 1 -- 172.21.15.88:0/2136594859 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74f400beb0 con 0x7f74f4005000 2023-10-18T16:07:35.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.664+0000 7f750cca9700 1 -- 172.21.15.88:0/2136594859 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74f400c070 con 0x7f74f4004690 2023-10-18T16:07:35.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.664+0000 7f750dcab700 1 --2- 172.21.15.88:0/2136594859 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74f40038e0 0x7f74f4003d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:35.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.664+0000 7f750e4ac700 1 --2- 172.21.15.88:0/2136594859 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f74f4005000 0x7f74f400b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:35.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.664+0000 7f750d4aa700 1 --2- 172.21.15.88:0/2136594859 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f74f4004690 0x7f74f4004b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:35.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.664+0000 7f750dcab700 1 -- 172.21.15.88:0/2136594859 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f74f4004690 msgr2=0x7f74f4004b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:35.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.664+0000 7f750dcab700 1 --2- 172.21.15.88:0/2136594859 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f74f4004690 0x7f74f4004b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:35.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.664+0000 7f750dcab700 1 -- 172.21.15.88:0/2136594859 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f74f4005000 msgr2=0x7f74f400b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:35.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.664+0000 7f750dcab700 1 --2- 172.21.15.88:0/2136594859 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f74f4005000 0x7f74f400b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:35.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.664+0000 7f750e4ac700 1 -- 172.21.15.88:0/2136594859 reap_dead start 2023-10-18T16:07:35.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.664+0000 7f750dcab700 1 -- 172.21.15.88:0/2136594859 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74f40033f0 con 0x7f74f40038e0 2023-10-18T16:07:35.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.664+0000 7f750d4aa700 1 --2- 172.21.15.88:0/2136594859 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f74f4004690 0x7f74f4004b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:07:35.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.664+0000 7f750dcab700 1 --2- 172.21.15.88:0/2136594859 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74f40038e0 0x7f74f4003d00 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f750400c190 tx=0x7f7508068d10).ready entity=mon.0 client_cookie=a550ef0db3bbb92a server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:35.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.664+0000 7f750cca9700 1 -- 172.21.15.88:0/2136594859 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75040070d0 con 0x7f74f40038e0 2023-10-18T16:07:35.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.664+0000 7f750cca9700 1 -- 172.21.15.88:0/2136594859 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7504007230 con 0x7f74f40038e0 2023-10-18T16:07:35.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.664+0000 7f750cca9700 1 -- 172.21.15.88:0/2136594859 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f750400ca40 con 0x7f74f40038e0 2023-10-18T16:07:35.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.665+0000 7f750ff0f700 1 -- 172.21.15.88:0/2136594859 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74f40038e0 msgr2=0x7f74f4003d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:35.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.665+0000 7f750ff0f700 1 --2- 172.21.15.88:0/2136594859 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74f40038e0 0x7f74f4003d00 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f750400c190 tx=0x7f7508068d10).stop 2023-10-18T16:07:35.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.665+0000 7f750ff0f700 1 -- 172.21.15.88:0/2136594859 shutdown_connections 2023-10-18T16:07:35.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.665+0000 7f750ff0f700 1 --2- 172.21.15.88:0/2136594859 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74f40038e0 0x7f74f4003d00 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:35.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.665+0000 7f750ff0f700 1 -- 172.21.15.88:0/2136594859 >> 172.21.15.88:0/2136594859 conn(0x7f7508064230 msgr2=0x7f75080c36c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:35.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.665+0000 7f750ff0f700 1 -- 172.21.15.88:0/2136594859 shutdown_connections 2023-10-18T16:07:35.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.665+0000 7f750ff0f700 1 -- 172.21.15.88:0/2136594859 wait complete. 2023-10-18T16:07:35.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.666+0000 7f750ff0f700 1 Processor -- start 2023-10-18T16:07:35.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.666+0000 7f750ff0f700 1 -- start start 2023-10-18T16:07:35.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.667+0000 7f750ff0f700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f74f40038e0 0x7f750814f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:35.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.667+0000 7f750ff0f700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f750814f9f0 0x7f7508160080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:35.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.667+0000 7f750ff0f700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7508160570 0x7f750813ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:35.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.667+0000 7f750ff0f700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f750814c4a0 con 0x7f7508160570 2023-10-18T16:07:35.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.667+0000 7f750ff0f700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7508149a10 con 0x7f750814f9f0 2023-10-18T16:07:35.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.667+0000 7f750ff0f700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f750814d440 con 0x7f74f40038e0 2023-10-18T16:07:35.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.667+0000 7f750e4ac700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7508160570 0x7f750813ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:35.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.667+0000 7f750e4ac700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7508160570 0x7f750813ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:34554/0 (socket says 172.21.15.88:34554) 2023-10-18T16:07:35.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.667+0000 7f750dcab700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f74f40038e0 0x7f750814f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:35.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.667+0000 7f750e4ac700 1 -- 172.21.15.88:0/2746402594 learned_addr learned my addr 172.21.15.88:0/2746402594 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:35.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.667+0000 7f750d4aa700 1 --2- 172.21.15.88:0/2746402594 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f750814f9f0 0x7f7508160080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:35.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.667+0000 7f750e4ac700 1 -- 172.21.15.88:0/2746402594 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f74f40038e0 msgr2=0x7f750814f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:35.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.667+0000 7f750e4ac700 1 --2- 172.21.15.88:0/2746402594 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f74f40038e0 0x7f750814f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:35.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.667+0000 7f750e4ac700 1 -- 172.21.15.88:0/2746402594 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f750814f9f0 msgr2=0x7f7508160080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:35.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.667+0000 7f750e4ac700 1 --2- 172.21.15.88:0/2746402594 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f750814f9f0 0x7f7508160080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:35.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.667+0000 7f750e4ac700 1 -- 172.21.15.88:0/2746402594 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f750815fb80 con 0x7f7508160570 2023-10-18T16:07:35.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.667+0000 7f750e4ac700 1 --2- 172.21.15.88:0/2746402594 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7508160570 0x7f750813ef10 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f75080c3f30 tx=0x7f750000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:35.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.667+0000 7f74fdffb700 1 -- 172.21.15.88:0/2746402594 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f750000e040 con 0x7f7508160570 2023-10-18T16:07:35.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.668+0000 7f74fdffb700 1 -- 172.21.15.88:0/2746402594 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f750000f040 con 0x7f7508160570 2023-10-18T16:07:35.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.668+0000 7f750ff0f700 1 -- 172.21.15.88:0/2746402594 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75080bf310 con 0x7f7508160570 2023-10-18T16:07:35.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.668+0000 7f74fdffb700 1 -- 172.21.15.88:0/2746402594 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f750000fd70 con 0x7f7508160570 2023-10-18T16:07:35.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.668+0000 7f750ff0f700 1 -- 172.21.15.88:0/2746402594 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f750814d9e0 con 0x7f7508160570 2023-10-18T16:07:35.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.669+0000 7f750ff0f700 1 -- 172.21.15.88:0/2746402594 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f750815fda0 con 0x7f7508160570 2023-10-18T16:07:35.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.669+0000 7f74fdffb700 1 -- 172.21.15.88:0/2746402594 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f7500015660 con 0x7f7508160570 2023-10-18T16:07:35.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.669+0000 7f74fdffb700 1 --2- 172.21.15.88:0/2746402594 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f74ec05f150 0x7f74ec061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:35.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.669+0000 7f74fdffb700 1 -- 172.21.15.88:0/2746402594 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 5258+0+0 (secure 0 0 0) 0x7f750007a850 con 0x7f7508160570 2023-10-18T16:07:35.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.670+0000 7f750dcab700 1 --2- 172.21.15.88:0/2746402594 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f74ec05f150 0x7f74ec061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:35.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.670+0000 7f750dcab700 1 --2- 172.21.15.88:0/2746402594 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f74ec05f150 0x7f74ec061610 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f7504000ef0 tx=0x7f7504015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:35.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.674+0000 7f74fdffb700 1 -- 172.21.15.88:0/2746402594 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f750815fda0 con 0x7f7508160570 2023-10-18T16:07:35.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:35.838+0000 7f750ff0f700 1 -- 172.21.15.88:0/2746402594 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f750815fd70 con 0x7f7508160570 2023-10-18T16:07:36.537 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.534+0000 7f74fdffb700 1 -- 172.21.15.88:0/2746402594 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v17) v1 ==== 173+0+0 (secure 0 0 0) 0x7f750004be80 con 0x7f7508160570 2023-10-18T16:07:36.540 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.537+0000 7f750ff0f700 1 -- 172.21.15.88:0/2746402594 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f74ec05f150 msgr2=0x7f74ec061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:36.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.537+0000 7f750ff0f700 1 --2- 172.21.15.88:0/2746402594 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f74ec05f150 0x7f74ec061610 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f7504000ef0 tx=0x7f7504015000).stop 2023-10-18T16:07:36.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.537+0000 7f750ff0f700 1 -- 172.21.15.88:0/2746402594 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7508160570 msgr2=0x7f750813ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:36.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.537+0000 7f750ff0f700 1 --2- 172.21.15.88:0/2746402594 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7508160570 0x7f750813ef10 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f75080c3f30 tx=0x7f750000b040).stop 2023-10-18T16:07:36.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.537+0000 7f750ff0f700 1 -- 172.21.15.88:0/2746402594 shutdown_connections 2023-10-18T16:07:36.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.537+0000 7f750ff0f700 1 --2- 172.21.15.88:0/2746402594 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f74ec05f150 0x7f74ec061610 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:36.542 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.537+0000 7f750ff0f700 1 --2- 172.21.15.88:0/2746402594 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7508160570 0x7f750813ef10 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:36.542 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.537+0000 7f750ff0f700 1 --2- 172.21.15.88:0/2746402594 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f74f40038e0 0x7f750814f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:36.542 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.537+0000 7f750ff0f700 1 --2- 172.21.15.88:0/2746402594 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f750814f9f0 0x7f7508160080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:36.542 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.537+0000 7f750ff0f700 1 -- 172.21.15.88:0/2746402594 >> 172.21.15.88:0/2746402594 conn(0x7f7508064230 msgr2=0x7f75080be540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:36.542 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.537+0000 7f750ff0f700 1 -- 172.21.15.88:0/2746402594 shutdown_connections 2023-10-18T16:07:36.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.537+0000 7f750ff0f700 1 -- 172.21.15.88:0/2746402594 wait complete. 2023-10-18T16:07:36.543 INFO:teuthology.orchestra.run.smithi088.stderr:enabled application 'rbd' on pool 'rbd' 2023-10-18T16:07:36.555 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-10-18T16:07:36.555 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-18T16:07:36.556 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-18T16:07:36.558 INFO:tasks.ceph.mds.a:Started 2023-10-18T16:07:36.559 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-18T16:07:36.559 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-18T16:07:36.561 INFO:tasks.ceph.mds.c:Started 2023-10-18T16:07:36.561 INFO:tasks.ceph.mds.e:Restarting daemon 2023-10-18T16:07:36.562 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2023-10-18T16:07:36.564 INFO:tasks.ceph.mds.e:Started 2023-10-18T16:07:36.564 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-18T16:07:36.564 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-18T16:07:36.567 INFO:tasks.ceph.mds.b:Started 2023-10-18T16:07:36.567 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-18T16:07:36.567 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-18T16:07:36.570 INFO:tasks.ceph.mds.d:Started 2023-10-18T16:07:36.570 INFO:tasks.ceph.mds.f:Restarting daemon 2023-10-18T16:07:36.570 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2023-10-18T16:07:36.572 INFO:tasks.ceph.mds.f:Started 2023-10-18T16:07:36.573 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2023-10-18T16:07:36.573 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-18T16:07:36.634 INFO:tasks.ceph.mds.a.smithi088.stdout:starting mds.a at 2023-10-18T16:07:36.635 INFO:tasks.ceph.mds.c.smithi088.stdout:starting mds.c at 2023-10-18T16:07:36.640 INFO:tasks.ceph.mds.e.smithi088.stdout:starting mds.e at 2023-10-18T16:07:36.649 INFO:tasks.ceph.mds.d.smithi151.stdout:starting mds.d at 2023-10-18T16:07:36.649 INFO:tasks.ceph.mds.b.smithi151.stdout:starting mds.b at 2023-10-18T16:07:36.649 INFO:tasks.ceph.mds.b.smithi151.stderr:2023-10-18T16:07:36.646+0000 7f22577b7700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2023-10-18T16:07:36.653 INFO:tasks.ceph.mds.f.smithi151.stdout:starting mds.f at 2023-10-18T16:07:36.698 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.694+0000 7f9103355700 1 Processor -- start 2023-10-18T16:07:36.698 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.695+0000 7f9103355700 1 -- start start 2023-10-18T16:07:36.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.695+0000 7f9103355700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f90fc154280 0x7f90fc150a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:36.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.695+0000 7f9103355700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90fc0a8570 con 0x7f90fc14d880 2023-10-18T16:07:36.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.695+0000 7f9103355700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90fc154730 con 0x7f90fc0c20a0 2023-10-18T16:07:36.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.695+0000 7f9103355700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90fc0a87c0 con 0x7f90fc154280 2023-10-18T16:07:36.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.696+0000 7f91010f1700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f90fc14d880 0x7f90fc0c1990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:50848/0 (socket says 172.21.15.88:50848) 2023-10-18T16:07:36.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.696+0000 7f91010f1700 1 -- 172.21.15.88:0/613244156 learned_addr learned my addr 172.21.15.88:0/613244156 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:07:36.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.696+0000 7f91018f2700 1 --2- 172.21.15.88:0/613244156 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f90fc154280 0x7f90fc150a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:36.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.696+0000 7f90f3fff700 1 -- 172.21.15.88:0/613244156 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1406674668 0 0) 0x7f90fc0a8570 con 0x7f90fc14d880 2023-10-18T16:07:36.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.696+0000 7f90f3fff700 1 -- 172.21.15.88:0/613244156 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90e80033f0 con 0x7f90fc14d880 2023-10-18T16:07:36.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.696+0000 7f90f3fff700 1 -- 172.21.15.88:0/613244156 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4112462749 0 0) 0x7f90fc154730 con 0x7f90fc0c20a0 2023-10-18T16:07:36.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.696+0000 7f90f3fff700 1 -- 172.21.15.88:0/613244156 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90fc0a8570 con 0x7f90fc0c20a0 2023-10-18T16:07:36.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.696+0000 7f91018f2700 1 -- 172.21.15.88:0/613244156 >> v1:172.21.15.151:6789/0 conn(0x7f90fc0c20a0 legacy=0x7f90fc150340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:36.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.696+0000 7f91018f2700 1 -- 172.21.15.88:0/613244156 >> v1:172.21.15.88:6789/0 conn(0x7f90fc14d880 legacy=0x7f90fc0c1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:36.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.696+0000 7f91018f2700 1 -- 172.21.15.88:0/613244156 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90fc0a8aa0 con 0x7f90fc154280 2023-10-18T16:07:36.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.696+0000 7f91018f2700 1 --2- 172.21.15.88:0/613244156 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f90fc154280 0x7f90fc150a50 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f90f4009e90 tx=0x7f90f4009780).ready entity=mon.2 client_cookie=a83bfad152b22d2b server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:36.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.696+0000 7f90f3fff700 1 -- 172.21.15.88:0/613244156 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f400e070 con 0x7f90fc154280 2023-10-18T16:07:36.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.696+0000 7f90f3fff700 1 -- 172.21.15.88:0/613244156 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90f400bab0 con 0x7f90fc154280 2023-10-18T16:07:36.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.696+0000 7f90f3fff700 1 -- 172.21.15.88:0/613244156 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f400be30 con 0x7f90fc154280 2023-10-18T16:07:36.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.697+0000 7f9103355700 1 -- 172.21.15.88:0/613244156 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f90fc154280 msgr2=0x7f90fc150a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:36.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.697+0000 7f9103355700 1 --2- 172.21.15.88:0/613244156 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f90fc154280 0x7f90fc150a50 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f90f4009e90 tx=0x7f90f4009780).stop 2023-10-18T16:07:36.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.697+0000 7f9103355700 1 -- 172.21.15.88:0/613244156 shutdown_connections 2023-10-18T16:07:36.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.697+0000 7f9103355700 1 --2- 172.21.15.88:0/613244156 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f90fc154280 0x7f90fc150a50 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:36.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.697+0000 7f9103355700 1 -- 172.21.15.88:0/613244156 >> 172.21.15.88:0/613244156 conn(0x7f90fc069d80 msgr2=0x7f90fc06a190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:36.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.697+0000 7f9103355700 1 -- 172.21.15.88:0/613244156 shutdown_connections 2023-10-18T16:07:36.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.697+0000 7f9103355700 1 -- 172.21.15.88:0/613244156 wait complete. 2023-10-18T16:07:36.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.698+0000 7f9103355700 1 Processor -- start 2023-10-18T16:07:36.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.698+0000 7f9103355700 1 -- start start 2023-10-18T16:07:36.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.699+0000 7f9103355700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90fc0c20a0 0x7f90fc0c0230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:36.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.699+0000 7f9103355700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f90fc14d880 0x7f90fc0c0720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:36.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.699+0000 7f9103355700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f90fc154280 0x7f90fc0bc170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:36.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.699+0000 7f9103355700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90fc0be440 con 0x7f90fc0c20a0 2023-10-18T16:07:36.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.699+0000 7f9103355700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90fc164400 con 0x7f90fc14d880 2023-10-18T16:07:36.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.699+0000 7f9103355700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90fc0ac900 con 0x7f90fc154280 2023-10-18T16:07:36.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.699+0000 7f91010f1700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90fc0c20a0 0x7f90fc0c0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:36.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.699+0000 7f91010f1700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90fc0c20a0 0x7f90fc0c0230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:34630/0 (socket says 172.21.15.88:34630) 2023-10-18T16:07:36.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.699+0000 7f91010f1700 1 -- 172.21.15.88:0/3714230561 learned_addr learned my addr 172.21.15.88:0/3714230561 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:36.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.699+0000 7f91018f2700 1 --2- 172.21.15.88:0/3714230561 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f90fc154280 0x7f90fc0bc170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:36.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.699+0000 7f91008f0700 1 --2- 172.21.15.88:0/3714230561 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f90fc14d880 0x7f90fc0c0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:36.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.699+0000 7f91010f1700 1 -- 172.21.15.88:0/3714230561 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f90fc154280 msgr2=0x7f90fc0bc170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:36.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.699+0000 7f91010f1700 1 --2- 172.21.15.88:0/3714230561 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f90fc154280 0x7f90fc0bc170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:36.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.699+0000 7f91010f1700 1 -- 172.21.15.88:0/3714230561 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f90fc14d880 msgr2=0x7f90fc0c0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:36.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.699+0000 7f91010f1700 1 --2- 172.21.15.88:0/3714230561 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f90fc14d880 0x7f90fc0c0720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:36.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.699+0000 7f91010f1700 1 -- 172.21.15.88:0/3714230561 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90f4009a20 con 0x7f90fc0c20a0 2023-10-18T16:07:36.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.699+0000 7f91018f2700 1 --2- 172.21.15.88:0/3714230561 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f90fc154280 0x7f90fc0bc170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-10-18T16:07:36.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.699+0000 7f91008f0700 1 --2- 172.21.15.88:0/3714230561 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f90fc14d880 0x7f90fc0c0720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-10-18T16:07:36.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.699+0000 7f91010f1700 1 --2- 172.21.15.88:0/3714230561 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90fc0c20a0 0x7f90fc0c0230 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f90f800c590 tx=0x7f90f8007d20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:36.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.700+0000 7f90f17fa700 1 -- 172.21.15.88:0/3714230561 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f80072e0 con 0x7f90fc0c20a0 2023-10-18T16:07:36.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.700+0000 7f90f17fa700 1 -- 172.21.15.88:0/3714230561 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90f8007440 con 0x7f90fc0c20a0 2023-10-18T16:07:36.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.700+0000 7f9103355700 1 -- 172.21.15.88:0/3714230561 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90fc0bbce0 con 0x7f90fc0c20a0 2023-10-18T16:07:36.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.700+0000 7f90f17fa700 1 -- 172.21.15.88:0/3714230561 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f80057a0 con 0x7f90fc0c20a0 2023-10-18T16:07:36.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.700+0000 7f9103355700 1 -- 172.21.15.88:0/3714230561 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90fc0ac620 con 0x7f90fc0c20a0 2023-10-18T16:07:36.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.701+0000 7f9103355700 1 -- 172.21.15.88:0/3714230561 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90fc0bbfc0 con 0x7f90fc0c20a0 2023-10-18T16:07:36.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.701+0000 7f90f17fa700 1 -- 172.21.15.88:0/3714230561 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f90f8005bd0 con 0x7f90fc0c20a0 2023-10-18T16:07:36.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.701+0000 7f90f17fa700 1 --2- 172.21.15.88:0/3714230561 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f90e005f120 0x7f90e00615e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:36.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.701+0000 7f90f17fa700 1 -- 172.21.15.88:0/3714230561 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5269+0+0 (secure 0 0 0) 0x7f90f8079880 con 0x7f90fc0c20a0 2023-10-18T16:07:36.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.702+0000 7f91008f0700 1 --2- 172.21.15.88:0/3714230561 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f90e005f120 0x7f90e00615e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:36.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.702+0000 7f91008f0700 1 --2- 172.21.15.88:0/3714230561 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f90e005f120 0x7f90e00615e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f90fc064050 tx=0x7f90ec009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:36.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.705+0000 7f90f17fa700 1 -- 172.21.15.88:0/3714230561 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f90f804aea0 con 0x7f90fc0c20a0 2023-10-18T16:07:36.875 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.872+0000 7f9103355700 1 -- 172.21.15.88:0/3714230561 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f90fc06c4e0 con 0x7f90fc0c20a0 2023-10-18T16:07:36.876 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.873+0000 7f90f17fa700 1 -- 172.21.15.88:0/3714230561 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+13722 (secure 0 0 0) 0x7f90f8018070 con 0x7f90fc0c20a0 2023-10-18T16:07:36.876 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:07:36.877 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":17,"fsid":"fb1f77e8-ead0-4f90-8128-fdf5137eb16f","created":"2023-10-18T16:07:23.662468+0000","modified":"2023-10-18T16:07:36.530182+0000","last_up_change":"2023-10-18T16:07:32.498203+0000","last_in_change":"2023-10-18T16:07:26.064784+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-10-18T16:07:32.536320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-18T16:07:34.769989+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"e421c20c-f7bc-4574-97a2-201bb4a16b4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":91007},{"type":"v1","addr":"172.21.15.88:6809","nonce":91007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":91007},{"type":"v1","addr":"172.21.15.88:6811","nonce":91007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":91007},{"type":"v1","addr":"172.21.15.88:6815","nonce":91007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":91007},{"type":"v1","addr":"172.21.15.88:6813","nonce":91007}]},"public_addr":"172.21.15.88:6809/91007","cluster_addr":"172.21.15.88:6811/91007","heartbeat_back_addr":"172.21.15.88:6815/91007","heartbeat_front_addr":"172.21.15.88:6813/91007","state":["exists","up"]},{"osd":1,"uuid":"9485c5ec-379a-4038-a2d8-1f1081fe2eea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":91009},{"type":"v1","addr":"172.21.15.88:6825","nonce":91009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":91009},{"type":"v1","addr":"172.21.15.88:6827","nonce":91009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":91009},{"type":"v1","addr":"172.21.15.88:6831","nonce":91009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":91009},{"type":"v1","addr":"172.21.15.88:6829","nonce":91009}]},"public_addr":"172.21.15.88:6825/91009","cluster_addr":"172.21.15.88:6827/91009","heartbeat_back_addr":"172.21.15.88:6831/91009","heartbeat_front_addr":"172.21.15.88:6829/91009","state":["exists","up"]},{"osd":2,"uuid":"671895e2-57f3-4f1a-ba4a-a580af1f3ec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":91005},{"type":"v1","addr":"172.21.15.88:6801","nonce":91005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":91005},{"type":"v1","addr":"172.21.15.88:6803","nonce":91005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":91005},{"type":"v1","addr":"172.21.15.88:6807","nonce":91005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":91005},{"type":"v1","addr":"172.21.15.88:6805","nonce":91005}]},"public_addr":"172.21.15.88:6801/91005","cluster_addr":"172.21.15.88:6803/91005","heartbeat_back_addr":"172.21.15.88:6807/91005","heartbeat_front_addr":"172.21.15.88:6805/91005","state":["exists","up"]},{"osd":3,"uuid":"0c4e6e7f-7d9c-4e65-beec-fd0ca384f320","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":91008},{"type":"v1","addr":"172.21.15.88:6817","nonce":91008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":91008},{"type":"v1","addr":"172.21.15.88:6819","nonce":91008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":91008},{"type":"v1","addr":"172.21.15.88:6823","nonce":91008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":91008},{"type":"v1","addr":"172.21.15.88:6821","nonce":91008}]},"public_addr":"172.21.15.88:6817/91008","cluster_addr":"172.21.15.88:6819/91008","heartbeat_back_addr":"172.21.15.88:6823/91008","heartbeat_front_addr":"172.21.15.88:6821/91008","state":["exists","up"]},{"osd":4,"uuid":"29570691-ef35-4b80-9247-aa3703877af4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":91514},{"type":"v1","addr":"172.21.15.151:6801","nonce":91514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":91514},{"type":"v1","addr":"172.21.15.151:6803","nonce":91514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":91514},{"type":"v1","addr":"172.21.15.151:6807","nonce":91514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":91514},{"type":"v1","addr":"172.21.15.151:6805","nonce":91514}]},"public_addr":"172.21.15.151:6801/91514","cluster_addr":"172.21.15.151:6803/91514","heartbeat_back_addr":"172.21.15.151:6807/91514","heartbeat_front_addr":"172.21.15.151:6805/91514","state":["exists","up"]},{"osd":5,"uuid":"26540b7b-032a-4b3b-bf6c-b131a449fa99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":91513},{"type":"v1","addr":"172.21.15.151:6809","nonce":91513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":91513},{"type":"v1","addr":"172.21.15.151:6811","nonce":91513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":91513},{"type":"v1","addr":"172.21.15.151:6815","nonce":91513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":91513},{"type":"v1","addr":"172.21.15.151:6813","nonce":91513}]},"public_addr":"172.21.15.151:6809/91513","cluster_addr":"172.21.15.151:6811/91513","heartbeat_back_addr":"172.21.15.151:6815/91513","heartbeat_front_addr":"172.21.15.151:6813/91513","state":["exists","up"]},{"osd":6,"uuid":"7af01f9c-90db-45fd-93c8-09517c9b0881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":91544},{"type":"v1","addr":"172.21.15.151:6817","nonce":91544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":91544},{"type":"v1","addr":"172.21.15.151:6819","nonce":91544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":91544},{"type":"v1","addr":"172.21.15.151:6823","nonce":91544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":91544},{"type":"v1","addr":"172.21.15.151:6821","nonce":91544}]},"public_addr":"172.21.15.151:6817/91544","cluster_addr":"172.21.15.151:6819/91544","heartbeat_back_addr":"172.21.15.151:6823/91544","heartbeat_front_addr":"172.21.15.151:6821/91544","state":["exists","up"]},{"osd":7,"uuid":"b527b0a5-4249-4854-a30f-bff897757239","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":91545},{"type":"v1","addr":"172.21.15.151:6825","nonce":91545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":91545},{"type":"v1","addr":"172.21.15.151:6827","nonce":91545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":91545},{"type":"v1","addr":"172.21.15.151:6831","nonce":91545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":91545},{"type":"v1","addr":"172.21.15.151:6829","nonce":91545}]},"public_addr":"172.21.15.151:6825/91545","cluster_addr":"172.21.15.151:6827/91545","heartbeat_back_addr":"172.21.15.151:6831/91545","heartbeat_front_addr":"172.21.15.151:6829/91545","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T16:07:31.216742+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T16:07:31.467229+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T16:07:31.220809+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T16:07:31.414999+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T16:07:31.188976+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T16:07:31.252153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T16:07:31.254324+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T16:07:31.203434+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-18T16:07:36.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.875+0000 7f9103355700 1 -- 172.21.15.88:0/3714230561 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f90e005f120 msgr2=0x7f90e00615e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:36.879 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.875+0000 7f9103355700 1 --2- 172.21.15.88:0/3714230561 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f90e005f120 0x7f90e00615e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f90fc064050 tx=0x7f90ec009040).stop 2023-10-18T16:07:36.879 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.875+0000 7f9103355700 1 -- 172.21.15.88:0/3714230561 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90fc0c20a0 msgr2=0x7f90fc0c0230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:36.879 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.875+0000 7f9103355700 1 --2- 172.21.15.88:0/3714230561 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90fc0c20a0 0x7f90fc0c0230 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f90f800c590 tx=0x7f90f8007d20).stop 2023-10-18T16:07:36.879 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.875+0000 7f9103355700 1 -- 172.21.15.88:0/3714230561 shutdown_connections 2023-10-18T16:07:36.879 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.875+0000 7f9103355700 1 --2- 172.21.15.88:0/3714230561 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f90e005f120 0x7f90e00615e0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:36.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.875+0000 7f9103355700 1 --2- 172.21.15.88:0/3714230561 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f90fc154280 0x7f90fc0bc170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:36.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.875+0000 7f9103355700 1 --2- 172.21.15.88:0/3714230561 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90fc0c20a0 0x7f90fc0c0230 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:36.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.875+0000 7f9103355700 1 --2- 172.21.15.88:0/3714230561 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f90fc14d880 0x7f90fc0c0720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:36.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.875+0000 7f9103355700 1 -- 172.21.15.88:0/3714230561 >> 172.21.15.88:0/3714230561 conn(0x7f90fc069d80 msgr2=0x7f90fc1517b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:36.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.875+0000 7f9103355700 1 -- 172.21.15.88:0/3714230561 shutdown_connections 2023-10-18T16:07:36.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:36.875+0000 7f9103355700 1 -- 172.21.15.88:0/3714230561 wait complete. 2023-10-18T16:07:36.890 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2023-10-18T16:07:32.536320+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-10-18T16:07:34.769989+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2023-10-18T16:07:36.891 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2023-10-18T16:07:37.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.003+0000 7f7ebdaa8700 1 Processor -- start 2023-10-18T16:07:37.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.003+0000 7f7ebdaa8700 1 -- start start 2023-10-18T16:07:37.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.004+0000 7f7ebdaa8700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7eb814db20 0x7f7eb8154430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:37.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.004+0000 7f7ebdaa8700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7eb80c22e0 0x7f7eb80a8650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:37.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.004+0000 7f7ebdaa8700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7eb80c31e0 con 0x7f7eb80c2760 2023-10-18T16:07:37.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.004+0000 7f7ebdaa8700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7eb80c1e50 con 0x7f7eb814db20 2023-10-18T16:07:37.010 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.004+0000 7f7ebdaa8700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb80a90b0 con 0x7f7eb80c22e0 2023-10-18T16:07:37.010 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.004+0000 7f7eb77fe700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7eb814db20 0x7f7eb8154430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:37.010 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.004+0000 7f7eb77fe700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7eb814db20 0x7f7eb8154430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:34640/0 (socket says 172.21.15.88:34640) 2023-10-18T16:07:37.010 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.004+0000 7f7eb77fe700 1 -- 172.21.15.88:0/3841245717 learned_addr learned my addr 172.21.15.88:0/3841245717 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:37.010 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.004+0000 7f7eb7fff700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f7eb80c2760 0x7f7eb80c34e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:50858/0 (socket says 172.21.15.88:50858) 2023-10-18T16:07:37.011 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.004+0000 7f7eb6ffd700 1 --2- 172.21.15.88:0/3841245717 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7eb80c22e0 0x7f7eb80a8650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:37.011 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.004+0000 7f7eb77fe700 1 -- 172.21.15.88:0/3841245717 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7eb80c22e0 msgr2=0x7f7eb80a8650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:37.011 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.004+0000 7f7eb77fe700 1 --2- 172.21.15.88:0/3841245717 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7eb80c22e0 0x7f7eb80a8650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:37.011 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.004+0000 7f7eb77fe700 1 -- 172.21.15.88:0/3841245717 >> v1:172.21.15.88:6789/0 conn(0x7f7eb80c2760 legacy=0x7f7eb80c34e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:37.011 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.004+0000 7f7eb77fe700 1 -- 172.21.15.88:0/3841245717 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eb80a9390 con 0x7f7eb814db20 2023-10-18T16:07:37.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.004+0000 7f7eb67fc700 1 -- 172.21.15.88:0/3841245717 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 125609115 0 0) 0x7f7eb0001110 con 0x7f7eb80c2760 2023-10-18T16:07:37.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.004+0000 7f7eb77fe700 1 --2- 172.21.15.88:0/3841245717 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7eb814db20 0x7f7eb8154430 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f7ea8009f20 tx=0x7f7ea8009840).ready entity=mon.0 client_cookie=8fb43b0d4453595a server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:37.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.004+0000 7f7eb67fc700 1 -- 172.21.15.88:0/3841245717 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ea800e070 con 0x7f7eb814db20 2023-10-18T16:07:37.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.005+0000 7f7eb67fc700 1 -- 172.21.15.88:0/3841245717 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ea800ba80 con 0x7f7eb814db20 2023-10-18T16:07:37.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.005+0000 7f7eb67fc700 1 -- 172.21.15.88:0/3841245717 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ea800be00 con 0x7f7eb814db20 2023-10-18T16:07:37.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.005+0000 7f7ebdaa8700 1 -- 172.21.15.88:0/3841245717 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7eb814db20 msgr2=0x7f7eb8154430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:37.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.005+0000 7f7ebdaa8700 1 --2- 172.21.15.88:0/3841245717 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7eb814db20 0x7f7eb8154430 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f7ea8009f20 tx=0x7f7ea8009840).stop 2023-10-18T16:07:37.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.005+0000 7f7ebdaa8700 1 -- 172.21.15.88:0/3841245717 shutdown_connections 2023-10-18T16:07:37.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.005+0000 7f7ebdaa8700 1 --2- 172.21.15.88:0/3841245717 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7eb814db20 0x7f7eb8154430 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:37.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.005+0000 7f7ebdaa8700 1 --2- 172.21.15.88:0/3841245717 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7eb80c22e0 0x7f7eb80a8650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:37.014 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.005+0000 7f7ebdaa8700 1 -- 172.21.15.88:0/3841245717 >> 172.21.15.88:0/3841245717 conn(0x7f7eb8069d80 msgr2=0x7f7eb806a190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:37.014 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.005+0000 7f7ebdaa8700 1 -- 172.21.15.88:0/3841245717 shutdown_connections 2023-10-18T16:07:37.014 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.005+0000 7f7ebdaa8700 1 -- 172.21.15.88:0/3841245717 wait complete. 2023-10-18T16:07:37.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.006+0000 7f7ebdaa8700 1 Processor -- start 2023-10-18T16:07:37.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.006+0000 7f7ebdaa8700 1 -- start start 2023-10-18T16:07:37.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.007+0000 7f7ebdaa8700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7eb80c22e0 0x7f7eb81605f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:37.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.007+0000 7f7ebdaa8700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7eb80c2760 0x7f7eb814c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:37.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.007+0000 7f7ebdaa8700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7eb814db20 0x7f7eb8150980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:37.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.007+0000 7f7ebdaa8700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb8150480 con 0x7f7eb80c2760 2023-10-18T16:07:37.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.007+0000 7f7ebdaa8700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb8163fe0 con 0x7f7eb80c22e0 2023-10-18T16:07:37.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.007+0000 7f7ebdaa8700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb81642c0 con 0x7f7eb814db20 2023-10-18T16:07:37.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.007+0000 7f7eb6ffd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7eb80c2760 0x7f7eb814c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:37.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.007+0000 7f7eb6ffd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7eb80c2760 0x7f7eb814c5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:34642/0 (socket says 172.21.15.88:34642) 2023-10-18T16:07:37.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.007+0000 7f7eb6ffd700 1 -- 172.21.15.88:0/1694557295 learned_addr learned my addr 172.21.15.88:0/1694557295 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:37.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.007+0000 7f7eb7fff700 1 --2- 172.21.15.88:0/1694557295 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7eb814db20 0x7f7eb8150980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:37.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.007+0000 7f7eb77fe700 1 --2- 172.21.15.88:0/1694557295 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7eb80c22e0 0x7f7eb81605f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:37.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.007+0000 7f7eb6ffd700 1 -- 172.21.15.88:0/1694557295 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7eb814db20 msgr2=0x7f7eb8150980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:37.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.007+0000 7f7eb6ffd700 1 --2- 172.21.15.88:0/1694557295 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7eb814db20 0x7f7eb8150980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:37.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.007+0000 7f7eb6ffd700 1 -- 172.21.15.88:0/1694557295 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7eb80c22e0 msgr2=0x7f7eb81605f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:37.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.007+0000 7f7eb6ffd700 1 --2- 172.21.15.88:0/1694557295 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7eb80c22e0 0x7f7eb81605f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:37.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.007+0000 7f7eb6ffd700 1 -- 172.21.15.88:0/1694557295 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ea8009ae0 con 0x7f7eb80c2760 2023-10-18T16:07:37.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.007+0000 7f7eb77fe700 1 --2- 172.21.15.88:0/1694557295 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7eb80c22e0 0x7f7eb81605f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:07:37.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.008+0000 7f7eb6ffd700 1 --2- 172.21.15.88:0/1694557295 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7eb80c2760 0x7f7eb814c5a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f7eac00d890 tx=0x7f7eac008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:37.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.008+0000 7f7e9f7fe700 1 -- 172.21.15.88:0/1694557295 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eac0125f0 con 0x7f7eb80c2760 2023-10-18T16:07:37.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.008+0000 7f7e9f7fe700 1 -- 172.21.15.88:0/1694557295 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7eac012750 con 0x7f7eb80c2760 2023-10-18T16:07:37.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.008+0000 7f7e9f7fe700 1 -- 172.21.15.88:0/1694557295 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eac012ad0 con 0x7f7eb80c2760 2023-10-18T16:07:37.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.008+0000 7f7ebdaa8700 1 -- 172.21.15.88:0/1694557295 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7eb81645a0 con 0x7f7eb80c2760 2023-10-18T16:07:37.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.008+0000 7f7ebdaa8700 1 -- 172.21.15.88:0/1694557295 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7eb80a9a40 con 0x7f7eb80c2760 2023-10-18T16:07:37.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.009+0000 7f7ebdaa8700 1 -- 172.21.15.88:0/1694557295 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7eb806c4e0 con 0x7f7eb80c2760 2023-10-18T16:07:37.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.009+0000 7f7e9f7fe700 1 -- 172.21.15.88:0/1694557295 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f7eac013480 con 0x7f7eb80c2760 2023-10-18T16:07:37.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.009+0000 7f7e9f7fe700 1 --2- 172.21.15.88:0/1694557295 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f7ea405f150 0x7f7ea4061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:37.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.009+0000 7f7e9f7fe700 1 -- 172.21.15.88:0/1694557295 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5269+0+0 (secure 0 0 0) 0x7f7eac079f50 con 0x7f7eb80c2760 2023-10-18T16:07:37.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.010+0000 7f7eb77fe700 1 --2- 172.21.15.88:0/1694557295 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f7ea405f150 0x7f7ea4061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:37.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.010+0000 7f7eb77fe700 1 --2- 172.21.15.88:0/1694557295 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f7ea405f150 0x7f7ea4061610 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f7ea800c010 tx=0x7f7ea8009840).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:37.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.014+0000 7f7e9f7fe700 1 -- 172.21.15.88:0/1694557295 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f7eac04b510 con 0x7f7eb80c2760 2023-10-18T16:07:37.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.176+0000 7f7ebdaa8700 1 -- 172.21.15.88:0/1694557295 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f7eb80656b0 con 0x7f7eb80c2760 2023-10-18T16:07:37.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.176+0000 7f7e9f7fe700 1 -- 172.21.15.88:0/1694557295 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v17) v1 ==== 110+0+10 (secure 0 0 0) 0x7f7eac04eb30 con 0x7f7eb80c2760 2023-10-18T16:07:37.180 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 1 2023-10-18T16:07:37.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.178+0000 7f7ebdaa8700 1 -- 172.21.15.88:0/1694557295 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f7ea405f150 msgr2=0x7f7ea4061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:37.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.178+0000 7f7ebdaa8700 1 --2- 172.21.15.88:0/1694557295 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f7ea405f150 0x7f7ea4061610 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f7ea800c010 tx=0x7f7ea8009840).stop 2023-10-18T16:07:37.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.178+0000 7f7ebdaa8700 1 -- 172.21.15.88:0/1694557295 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7eb80c2760 msgr2=0x7f7eb814c5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:37.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.178+0000 7f7ebdaa8700 1 --2- 172.21.15.88:0/1694557295 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7eb80c2760 0x7f7eb814c5a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f7eac00d890 tx=0x7f7eac008040).stop 2023-10-18T16:07:37.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.178+0000 7f7ebdaa8700 1 -- 172.21.15.88:0/1694557295 shutdown_connections 2023-10-18T16:07:37.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.178+0000 7f7ebdaa8700 1 --2- 172.21.15.88:0/1694557295 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f7ea405f150 0x7f7ea4061610 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:37.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.178+0000 7f7ebdaa8700 1 --2- 172.21.15.88:0/1694557295 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7eb814db20 0x7f7eb8150980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:37.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.178+0000 7f7ebdaa8700 1 --2- 172.21.15.88:0/1694557295 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7eb80c22e0 0x7f7eb81605f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:37.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.178+0000 7f7ebdaa8700 1 --2- 172.21.15.88:0/1694557295 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7eb80c2760 0x7f7eb814c5a0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:37.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.178+0000 7f7ebdaa8700 1 -- 172.21.15.88:0/1694557295 >> 172.21.15.88:0/1694557295 conn(0x7f7eb8069d80 msgr2=0x7f7eb80c55b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:37.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.178+0000 7f7ebdaa8700 1 -- 172.21.15.88:0/1694557295 shutdown_connections 2023-10-18T16:07:37.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.178+0000 7f7ebdaa8700 1 -- 172.21.15.88:0/1694557295 wait complete. 2023-10-18T16:07:37.193 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-10-18T16:07:37.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.308+0000 7ff3d4bbc700 1 Processor -- start 2023-10-18T16:07:37.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.308+0000 7ff3d4bbc700 1 -- start start 2023-10-18T16:07:37.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.309+0000 7ff3d4bbc700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff3d0068230 0x7ff3d00c0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:37.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.309+0000 7ff3d4bbc700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7ff3d014d990 0x7ff3d0149e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:37.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.309+0000 7ff3d4bbc700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3d006a700 con 0x7ff3d00c0360 2023-10-18T16:07:37.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.309+0000 7ff3d4bbc700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3d0069b80 con 0x7ff3d014d990 2023-10-18T16:07:37.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.309+0000 7ff3d4bbc700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d006ab90 con 0x7ff3d0068230 2023-10-18T16:07:37.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.309+0000 7ff3ce59c700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7ff3d00c0360 0x7ff3d00be110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:50872/0 (socket says 172.21.15.88:50872) 2023-10-18T16:07:37.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.309+0000 7ff3ce59c700 1 -- 172.21.15.88:0/1922444321 learned_addr learned my addr 172.21.15.88:0/1922444321 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:07:37.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.309+0000 7ff3ced9d700 1 --2- 172.21.15.88:0/1922444321 >> v2:172.21.15.151:3300/0 conn(0x7ff3d014d990 0x7ff3d0149e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:37.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.309+0000 7ff3cdd9b700 1 --2- 172.21.15.88:0/1922444321 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff3d0068230 0x7ff3d00c0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:37.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.309+0000 7ff3cd59a700 1 -- 172.21.15.88:0/1922444321 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 831160760 0 0) 0x7ff3d006a700 con 0x7ff3d00c0360 2023-10-18T16:07:37.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.310+0000 7ff3cd59a700 1 -- 172.21.15.88:0/1922444321 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3b80033f0 con 0x7ff3d00c0360 2023-10-18T16:07:37.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.310+0000 7ff3cd59a700 1 -- 172.21.15.88:0/1922444321 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2501586729 0 0) 0x7ff3b80033f0 con 0x7ff3d00c0360 2023-10-18T16:07:37.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.310+0000 7ff3cd59a700 1 -- 172.21.15.88:0/1922444321 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff3d0068230 msgr2=0x7ff3d00c0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:37.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.310+0000 7ff3cd59a700 1 --2- 172.21.15.88:0/1922444321 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff3d0068230 0x7ff3d00c0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:37.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.310+0000 7ff3cd59a700 1 -- 172.21.15.88:0/1922444321 >> v2:172.21.15.151:3300/0 conn(0x7ff3d014d990 msgr2=0x7ff3d0149e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:37.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.310+0000 7ff3cd59a700 1 --2- 172.21.15.88:0/1922444321 >> v2:172.21.15.151:3300/0 conn(0x7ff3d014d990 0x7ff3d0149e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:37.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.310+0000 7ff3cd59a700 1 -- 172.21.15.88:0/1922444321 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3d014de00 con 0x7ff3d00c0360 2023-10-18T16:07:37.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.310+0000 7ff3cd59a700 1 -- 172.21.15.88:0/1922444321 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441715075 0 0) 0x7ff3c00017f0 con 0x7ff3d00c0360 2023-10-18T16:07:37.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.310+0000 7ff3cd59a700 1 -- 172.21.15.88:0/1922444321 >> v1:172.21.15.88:6789/0 conn(0x7ff3d00c0360 legacy=0x7ff3d00be110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:37.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.310+0000 7ff3cd59a700 1 --2- 172.21.15.88:0/1922444321 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff3d0068230 0x7ff3b80038e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:37.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.310+0000 7ff3cd59a700 1 --2- 172.21.15.88:0/1922444321 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff3b8003e10 0x7ff3b8004690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:37.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.310+0000 7ff3cd59a700 1 --2- 172.21.15.88:0/1922444321 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3b8004b80 0x7ff3b8005000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:37.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.310+0000 7ff3cd59a700 1 -- 172.21.15.88:0/1922444321 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d0069b80 con 0x7ff3b8004b80 2023-10-18T16:07:37.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.310+0000 7ff3cd59a700 1 -- 172.21.15.88:0/1922444321 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d006ab90 con 0x7ff3b8003e10 2023-10-18T16:07:37.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.310+0000 7ff3cd59a700 1 -- 172.21.15.88:0/1922444321 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b800b8e0 con 0x7ff3d0068230 2023-10-18T16:07:37.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.310+0000 7ff3ced9d700 1 --2- 172.21.15.88:0/1922444321 >> v2:172.21.15.151:3300/0 conn(0x7ff3d014d990 0x7ff3d0149e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2023-10-18T16:07:37.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.310+0000 7ff3ced9d700 1 --2- 172.21.15.88:0/1922444321 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3b8004b80 0x7ff3b8005000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:37.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.310+0000 7ff3ce59c700 1 --2- 172.21.15.88:0/1922444321 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff3d0068230 0x7ff3b80038e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:37.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.310+0000 7ff3cdd9b700 1 --2- 172.21.15.88:0/1922444321 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff3b8003e10 0x7ff3b8004690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:37.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.310+0000 7ff3ced9d700 1 -- 172.21.15.88:0/1922444321 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff3d0068230 msgr2=0x7ff3b80038e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:37.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.310+0000 7ff3ced9d700 1 --2- 172.21.15.88:0/1922444321 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff3d0068230 0x7ff3b80038e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:37.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.310+0000 7ff3ced9d700 1 -- 172.21.15.88:0/1922444321 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff3b8003e10 msgr2=0x7ff3b8004690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:37.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.310+0000 7ff3ced9d700 1 --2- 172.21.15.88:0/1922444321 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff3b8003e10 0x7ff3b8004690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:37.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.310+0000 7ff3ced9d700 1 -- 172.21.15.88:0/1922444321 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3b80033f0 con 0x7ff3b8004b80 2023-10-18T16:07:37.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.310+0000 7ff3ced9d700 1 --2- 172.21.15.88:0/1922444321 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3b8004b80 0x7ff3b8005000 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7ff3c8009c60 tx=0x7ff3c8009060).ready entity=mon.0 client_cookie=1a26bafa60e09863 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:37.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.310+0000 7ff3cd59a700 1 -- 172.21.15.88:0/1922444321 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c80095c0 con 0x7ff3b8004b80 2023-10-18T16:07:37.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.310+0000 7ff3cd59a700 1 -- 172.21.15.88:0/1922444321 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3c800e710 con 0x7ff3b8004b80 2023-10-18T16:07:37.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.310+0000 7ff3cd59a700 1 -- 172.21.15.88:0/1922444321 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c800ea90 con 0x7ff3b8004b80 2023-10-18T16:07:37.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.311+0000 7ff3d4bbc700 1 -- 172.21.15.88:0/1922444321 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3b8004b80 msgr2=0x7ff3b8005000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:37.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.311+0000 7ff3d4bbc700 1 --2- 172.21.15.88:0/1922444321 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3b8004b80 0x7ff3b8005000 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7ff3c8009c60 tx=0x7ff3c8009060).stop 2023-10-18T16:07:37.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.311+0000 7ff3ced9d700 1 -- 172.21.15.88:0/1922444321 reap_dead start 2023-10-18T16:07:37.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.311+0000 7ff3d4bbc700 1 -- 172.21.15.88:0/1922444321 shutdown_connections 2023-10-18T16:07:37.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.311+0000 7ff3d4bbc700 1 -- 172.21.15.88:0/1922444321 >> 172.21.15.88:0/1922444321 conn(0x7ff3d0063760 msgr2=0x7ff3d0063b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:37.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.311+0000 7ff3d4bbc700 1 -- 172.21.15.88:0/1922444321 shutdown_connections 2023-10-18T16:07:37.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.311+0000 7ff3d4bbc700 1 -- 172.21.15.88:0/1922444321 wait complete. 2023-10-18T16:07:37.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.312+0000 7ff3d4bbc700 1 Processor -- start 2023-10-18T16:07:37.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.312+0000 7ff3d4bbc700 1 -- start start 2023-10-18T16:07:37.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.313+0000 7ff3d4bbc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff3b8004b80 0x7ff3d0062810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:37.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.313+0000 7ff3d4bbc700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3d0062d00 0x7ff3d00b8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:37.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.313+0000 7ff3d4bbc700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff3d0150c20 0x7ff3d00b94a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:37.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.313+0000 7ff3d4bbc700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d0164680 con 0x7ff3d0062d00 2023-10-18T16:07:37.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.313+0000 7ff3d4bbc700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d0151110 con 0x7ff3b8004b80 2023-10-18T16:07:37.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.313+0000 7ff3d4bbc700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d0163d50 con 0x7ff3d0150c20 2023-10-18T16:07:37.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.313+0000 7ff3cdd9b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3d0062d00 0x7ff3d00b8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:37.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.313+0000 7ff3cdd9b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3d0062d00 0x7ff3d00b8fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:34666/0 (socket says 172.21.15.88:34666) 2023-10-18T16:07:37.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.313+0000 7ff3cdd9b700 1 -- 172.21.15.88:0/557397288 learned_addr learned my addr 172.21.15.88:0/557397288 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:37.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.313+0000 7ff3ce59c700 1 --2- 172.21.15.88:0/557397288 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff3b8004b80 0x7ff3d0062810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:37.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.313+0000 7ff3ced9d700 1 --2- 172.21.15.88:0/557397288 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff3d0150c20 0x7ff3d00b94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:37.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.313+0000 7ff3cdd9b700 1 -- 172.21.15.88:0/557397288 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff3d0150c20 msgr2=0x7ff3d00b94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:37.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.313+0000 7ff3cdd9b700 1 --2- 172.21.15.88:0/557397288 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff3d0150c20 0x7ff3d00b94a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:37.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.313+0000 7ff3cdd9b700 1 -- 172.21.15.88:0/557397288 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff3b8004b80 msgr2=0x7ff3d0062810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:37.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.313+0000 7ff3cdd9b700 1 --2- 172.21.15.88:0/557397288 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff3b8004b80 0x7ff3d0062810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:37.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.313+0000 7ff3cdd9b700 1 -- 172.21.15.88:0/557397288 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3d0164030 con 0x7ff3d0062d00 2023-10-18T16:07:37.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.313+0000 7ff3ced9d700 1 --2- 172.21.15.88:0/557397288 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff3d0150c20 0x7ff3d00b94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:07:37.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.313+0000 7ff3cdd9b700 1 --2- 172.21.15.88:0/557397288 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3d0062d00 0x7ff3d00b8fb0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7ff3c40119c0 tx=0x7ff3c4009060).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:37.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.313+0000 7ff3be7fc700 1 -- 172.21.15.88:0/557397288 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c4011eb0 con 0x7ff3d0062d00 2023-10-18T16:07:37.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.313+0000 7ff3be7fc700 1 -- 172.21.15.88:0/557397288 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3c400f970 con 0x7ff3d0062d00 2023-10-18T16:07:37.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.313+0000 7ff3d4bbc700 1 -- 172.21.15.88:0/557397288 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3d0164310 con 0x7ff3d0062d00 2023-10-18T16:07:37.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.313+0000 7ff3be7fc700 1 -- 172.21.15.88:0/557397288 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c400fcf0 con 0x7ff3d0062d00 2023-10-18T16:07:37.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.313+0000 7ff3d4bbc700 1 -- 172.21.15.88:0/557397288 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3d00b6ef0 con 0x7ff3d0062d00 2023-10-18T16:07:37.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.314+0000 7ff3d4bbc700 1 -- 172.21.15.88:0/557397288 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3d006c4e0 con 0x7ff3d0062d00 2023-10-18T16:07:37.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.315+0000 7ff3be7fc700 1 -- 172.21.15.88:0/557397288 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7ff3c4014650 con 0x7ff3d0062d00 2023-10-18T16:07:37.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.315+0000 7ff3be7fc700 1 --2- 172.21.15.88:0/557397288 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7ff3b005f150 0x7ff3b0061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:37.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.315+0000 7ff3be7fc700 1 -- 172.21.15.88:0/557397288 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5269+0+0 (secure 0 0 0) 0x7ff3c407a440 con 0x7ff3d0062d00 2023-10-18T16:07:37.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.315+0000 7ff3ce59c700 1 --2- 172.21.15.88:0/557397288 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7ff3b005f150 0x7ff3b0061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:37.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.316+0000 7ff3ce59c700 1 --2- 172.21.15.88:0/557397288 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7ff3b005f150 0x7ff3b0061610 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7ff3c0007c70 tx=0x7ff3d00bb780).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:37.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.319+0000 7ff3be7fc700 1 -- 172.21.15.88:0/557397288 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7ff3c404ba60 con 0x7ff3d0062d00 2023-10-18T16:07:37.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.483+0000 7ff3d4bbc700 1 -- 172.21.15.88:0/557397288 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7ff3d01647c0 con 0x7ff3d0062d00 2023-10-18T16:07:37.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.483+0000 7ff3be7fc700 1 -- 172.21.15.88:0/557397288 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v17) v1 ==== 92+0+10 (secure 0 0 0) 0x7ff3c404f080 con 0x7ff3d0062d00 2023-10-18T16:07:37.488 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 8 2023-10-18T16:07:37.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.485+0000 7ff3d4bbc700 1 -- 172.21.15.88:0/557397288 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7ff3b005f150 msgr2=0x7ff3b0061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:37.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.485+0000 7ff3d4bbc700 1 --2- 172.21.15.88:0/557397288 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7ff3b005f150 0x7ff3b0061610 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7ff3c0007c70 tx=0x7ff3d00bb780).stop 2023-10-18T16:07:37.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.485+0000 7ff3d4bbc700 1 -- 172.21.15.88:0/557397288 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3d0062d00 msgr2=0x7ff3d00b8fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:37.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.485+0000 7ff3d4bbc700 1 --2- 172.21.15.88:0/557397288 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3d0062d00 0x7ff3d00b8fb0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7ff3c40119c0 tx=0x7ff3c4009060).stop 2023-10-18T16:07:37.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.485+0000 7ff3d4bbc700 1 -- 172.21.15.88:0/557397288 shutdown_connections 2023-10-18T16:07:37.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.485+0000 7ff3d4bbc700 1 --2- 172.21.15.88:0/557397288 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7ff3b005f150 0x7ff3b0061610 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:37.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.485+0000 7ff3d4bbc700 1 --2- 172.21.15.88:0/557397288 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff3d0150c20 0x7ff3d00b94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:37.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.485+0000 7ff3d4bbc700 1 --2- 172.21.15.88:0/557397288 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff3b8004b80 0x7ff3d0062810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:37.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.485+0000 7ff3d4bbc700 1 --2- 172.21.15.88:0/557397288 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3d0062d00 0x7ff3d00b8fb0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:37.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.485+0000 7ff3d4bbc700 1 -- 172.21.15.88:0/557397288 >> 172.21.15.88:0/557397288 conn(0x7ff3d0063760 msgr2=0x7ff3d014cfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:37.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.485+0000 7ff3d4bbc700 1 -- 172.21.15.88:0/557397288 shutdown_connections 2023-10-18T16:07:37.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.485+0000 7ff3d4bbc700 1 -- 172.21.15.88:0/557397288 wait complete. 2023-10-18T16:07:37.500 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-18T16:07:37.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.610+0000 7f0a48bdb700 1 Processor -- start 2023-10-18T16:07:37.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.610+0000 7f0a48bdb700 1 -- start start 2023-10-18T16:07:37.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.611+0000 7f0a48bdb700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0a44064e10 0x7f0a44063590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:37.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.611+0000 7f0a48bdb700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f0a44063a80 0x7f0a4414a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:37.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.611+0000 7f0a48bdb700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a441514e0 con 0x7f0a44064a30 2023-10-18T16:07:37.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.611+0000 7f0a48bdb700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a4414d280 con 0x7f0a44064e10 2023-10-18T16:07:37.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.611+0000 7f0a48bdb700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a440be5b0 con 0x7f0a44063a80 2023-10-18T16:07:37.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.611+0000 7f0a41d9b700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0a44064e10 0x7f0a44063590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:37.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.611+0000 7f0a4259c700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f0a44064a30 0x7f0a4414d710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:52338/0 (socket says 172.21.15.88:52338) 2023-10-18T16:07:37.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.611+0000 7f0a4259c700 1 -- 172.21.15.88:0/853304549 learned_addr learned my addr 172.21.15.88:0/853304549 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:07:37.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.612+0000 7f0a42d9d700 1 --2- 172.21.15.88:0/853304549 >> v2:172.21.15.151:3300/0 conn(0x7f0a44063a80 0x7f0a4414a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:37.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.612+0000 7f0a41d9b700 1 -- 172.21.15.88:0/853304549 >> v2:172.21.15.151:3300/0 conn(0x7f0a44063a80 msgr2=0x7f0a4414a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:37.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.612+0000 7f0a41d9b700 1 --2- 172.21.15.88:0/853304549 >> v2:172.21.15.151:3300/0 conn(0x7f0a44063a80 0x7f0a4414a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:37.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.612+0000 7f0a41d9b700 1 -- 172.21.15.88:0/853304549 >> v1:172.21.15.88:6789/0 conn(0x7f0a44064a30 legacy=0x7f0a4414d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:37.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.612+0000 7f0a41d9b700 1 -- 172.21.15.88:0/853304549 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a44150980 con 0x7f0a44064e10 2023-10-18T16:07:37.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.612+0000 7f0a4159a700 1 -- 172.21.15.88:0/853304549 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3375670722 0 0) 0x7f0a441514e0 con 0x7f0a44064a30 2023-10-18T16:07:37.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.612+0000 7f0a41d9b700 1 --2- 172.21.15.88:0/853304549 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a44064e10 0x7f0a44063590 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f0a44063310 tx=0x7f0a38009060).ready entity=mon.0 client_cookie=e86c4b1d19c14ded server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:37.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.612+0000 7f0a4159a700 1 -- 172.21.15.88:0/853304549 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a3800b580 con 0x7f0a44064e10 2023-10-18T16:07:37.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.612+0000 7f0a4159a700 1 -- 172.21.15.88:0/853304549 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a3800b6e0 con 0x7f0a44064e10 2023-10-18T16:07:37.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.612+0000 7f0a4159a700 1 -- 172.21.15.88:0/853304549 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a3800ba60 con 0x7f0a44064e10 2023-10-18T16:07:37.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.612+0000 7f0a48bdb700 1 -- 172.21.15.88:0/853304549 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a44064e10 msgr2=0x7f0a44063590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:37.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.612+0000 7f0a48bdb700 1 --2- 172.21.15.88:0/853304549 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a44064e10 0x7f0a44063590 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f0a44063310 tx=0x7f0a38009060).stop 2023-10-18T16:07:37.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.612+0000 7f0a48bdb700 1 -- 172.21.15.88:0/853304549 shutdown_connections 2023-10-18T16:07:37.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.612+0000 7f0a48bdb700 1 --2- 172.21.15.88:0/853304549 >> v2:172.21.15.151:3300/0 conn(0x7f0a44063a80 0x7f0a4414a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:37.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.612+0000 7f0a48bdb700 1 --2- 172.21.15.88:0/853304549 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a44064e10 0x7f0a44063590 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:37.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.612+0000 7f0a48bdb700 1 -- 172.21.15.88:0/853304549 >> 172.21.15.88:0/853304549 conn(0x7f0a44069360 msgr2=0x7f0a44069770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:37.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.613+0000 7f0a48bdb700 1 -- 172.21.15.88:0/853304549 shutdown_connections 2023-10-18T16:07:37.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.613+0000 7f0a48bdb700 1 -- 172.21.15.88:0/853304549 wait complete. 2023-10-18T16:07:37.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.614+0000 7f0a48bdb700 1 Processor -- start 2023-10-18T16:07:37.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.614+0000 7f0a48bdb700 1 -- start start 2023-10-18T16:07:37.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.614+0000 7f0a48bdb700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0a44063a80 0x7f0a4414feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:37.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.614+0000 7f0a48bdb700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a44064a30 0x7f0a44145bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:37.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.614+0000 7f0a48bdb700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0a44064e10 0x7f0a44160830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:37.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.614+0000 7f0a48bdb700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a44160330 con 0x7f0a44064a30 2023-10-18T16:07:37.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.614+0000 7f0a48bdb700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a44163a80 con 0x7f0a44063a80 2023-10-18T16:07:37.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.614+0000 7f0a48bdb700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a44163d60 con 0x7f0a44064e10 2023-10-18T16:07:37.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.614+0000 7f0a41d9b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a44064a30 0x7f0a44145bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:37.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.614+0000 7f0a41d9b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a44064a30 0x7f0a44145bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43842/0 (socket says 172.21.15.88:43842) 2023-10-18T16:07:37.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.614+0000 7f0a41d9b700 1 -- 172.21.15.88:0/1241526435 learned_addr learned my addr 172.21.15.88:0/1241526435 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:37.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.614+0000 7f0a42d9d700 1 --2- 172.21.15.88:0/1241526435 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0a44064e10 0x7f0a44160830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:37.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.614+0000 7f0a4259c700 1 --2- 172.21.15.88:0/1241526435 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0a44063a80 0x7f0a4414feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:37.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.615+0000 7f0a41d9b700 1 -- 172.21.15.88:0/1241526435 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0a44064e10 msgr2=0x7f0a44160830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:37.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.615+0000 7f0a41d9b700 1 --2- 172.21.15.88:0/1241526435 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0a44064e10 0x7f0a44160830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:37.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.615+0000 7f0a41d9b700 1 -- 172.21.15.88:0/1241526435 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0a44063a80 msgr2=0x7f0a4414feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:37.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.615+0000 7f0a41d9b700 1 --2- 172.21.15.88:0/1241526435 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0a44063a80 0x7f0a4414feb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:37.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.615+0000 7f0a41d9b700 1 -- 172.21.15.88:0/1241526435 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a380092a0 con 0x7f0a44064a30 2023-10-18T16:07:37.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.615+0000 7f0a4259c700 1 --2- 172.21.15.88:0/1241526435 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0a44063a80 0x7f0a4414feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:07:37.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.615+0000 7f0a42d9d700 1 --2- 172.21.15.88:0/1241526435 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0a44064e10 0x7f0a44160830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:07:37.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.615+0000 7f0a41d9b700 1 --2- 172.21.15.88:0/1241526435 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a44064a30 0x7f0a44145bb0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f0a38009fd0 tx=0x7f0a38017000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:37.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.615+0000 7f0a327fc700 1 -- 172.21.15.88:0/1241526435 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a3800b850 con 0x7f0a44064a30 2023-10-18T16:07:37.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.615+0000 7f0a327fc700 1 -- 172.21.15.88:0/1241526435 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a3800b9b0 con 0x7f0a44064a30 2023-10-18T16:07:37.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.615+0000 7f0a48bdb700 1 -- 172.21.15.88:0/1241526435 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a44164040 con 0x7f0a44064a30 2023-10-18T16:07:37.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.615+0000 7f0a327fc700 1 -- 172.21.15.88:0/1241526435 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a3801b620 con 0x7f0a44064a30 2023-10-18T16:07:37.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.615+0000 7f0a48bdb700 1 -- 172.21.15.88:0/1241526435 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a44164600 con 0x7f0a44064a30 2023-10-18T16:07:37.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.616+0000 7f0a48bdb700 1 -- 172.21.15.88:0/1241526435 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a4406c4e0 con 0x7f0a44064a30 2023-10-18T16:07:37.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.616+0000 7f0a327fc700 1 -- 172.21.15.88:0/1241526435 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f0a3801ba50 con 0x7f0a44064a30 2023-10-18T16:07:37.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.617+0000 7f0a327fc700 1 --2- 172.21.15.88:0/1241526435 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f0a2405f150 0x7f0a24061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:37.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.617+0000 7f0a327fc700 1 -- 172.21.15.88:0/1241526435 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5269+0+0 (secure 0 0 0) 0x7f0a3807b920 con 0x7f0a44064a30 2023-10-18T16:07:37.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.617+0000 7f0a4259c700 1 --2- 172.21.15.88:0/1241526435 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f0a2405f150 0x7f0a24061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:37.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.617+0000 7f0a4259c700 1 --2- 172.21.15.88:0/1241526435 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f0a2405f150 0x7f0a24061610 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f0a34007fd0 tx=0x7f0a34009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:37.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.621+0000 7f0a327fc700 1 -- 172.21.15.88:0/1241526435 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f0a3804cf40 con 0x7f0a44064a30 2023-10-18T16:07:37.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.827+0000 7f0a48bdb700 1 -- 172.21.15.88:0/1241526435 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0a44063370 con 0x7f0a44064a30 2023-10-18T16:07:37.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.828+0000 7f0a327fc700 1 -- 172.21.15.88:0/1241526435 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+5042 (secure 0 0 0) 0x7f0a38029090 con 0x7f0a44064a30 2023-10-18T16:07:37.832 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:07:37.832 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/3094207741","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3094207741},{"type":"v1","addr":"172.21.15.88:6833","nonce":3094207741}]},"join_fscid":-1,"export_targets":[],"features":4540138314316775423,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4272,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/1543332826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":1543332826},{"type":"v1","addr":"172.21.15.88:6835","nonce":1543332826}]},"join_fscid":-1,"export_targets":[],"features":4540138314316775423,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4279,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6836/941009236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6835","nonce":941009236},{"type":"v1","addr":"172.21.15.151:6836","nonce":941009236}]},"join_fscid":-1,"export_targets":[],"features":4540138314316775423,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/2987875245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2987875245},{"type":"v1","addr":"172.21.15.88:6837","nonce":2987875245}]},"join_fscid":-1,"export_targets":[],"features":4540138314316775423,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4288,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6839/2615482496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6838","nonce":2615482496},{"type":"v1","addr":"172.21.15.151:6839","nonce":2615482496}]},"join_fscid":-1,"export_targets":[],"features":4540138314316775423,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4293,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6837/3689661027","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6834","nonce":3689661027},{"type":"v1","addr":"172.21.15.151:6837","nonce":3689661027}]},"join_fscid":-1,"export_targets":[],"features":4540138314316775423,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2023-10-18T16:07:37.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.829+0000 7f0a48bdb700 1 -- 172.21.15.88:0/1241526435 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f0a2405f150 msgr2=0x7f0a24061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:37.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.829+0000 7f0a48bdb700 1 --2- 172.21.15.88:0/1241526435 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f0a2405f150 0x7f0a24061610 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f0a34007fd0 tx=0x7f0a34009040).stop 2023-10-18T16:07:37.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.829+0000 7f0a48bdb700 1 -- 172.21.15.88:0/1241526435 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a44064a30 msgr2=0x7f0a44145bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:37.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.829+0000 7f0a48bdb700 1 --2- 172.21.15.88:0/1241526435 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a44064a30 0x7f0a44145bb0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f0a38009fd0 tx=0x7f0a38017000).stop 2023-10-18T16:07:37.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.829+0000 7f0a48bdb700 1 -- 172.21.15.88:0/1241526435 shutdown_connections 2023-10-18T16:07:37.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.829+0000 7f0a48bdb700 1 --2- 172.21.15.88:0/1241526435 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f0a2405f150 0x7f0a24061610 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:37.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.829+0000 7f0a48bdb700 1 --2- 172.21.15.88:0/1241526435 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0a44064e10 0x7f0a44160830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:37.835 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.829+0000 7f0a48bdb700 1 --2- 172.21.15.88:0/1241526435 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0a44063a80 0x7f0a4414feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:37.835 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.829+0000 7f0a48bdb700 1 --2- 172.21.15.88:0/1241526435 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a44064a30 0x7f0a44145bb0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:37.835 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.829+0000 7f0a48bdb700 1 -- 172.21.15.88:0/1241526435 >> 172.21.15.88:0/1241526435 conn(0x7f0a44069360 msgr2=0x7f0a440bcba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:37.835 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.829+0000 7f0a48bdb700 1 -- 172.21.15.88:0/1241526435 shutdown_connections 2023-10-18T16:07:37.835 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.829+0000 7f0a48bdb700 1 -- 172.21.15.88:0/1241526435 wait complete. 2023-10-18T16:07:37.836 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 3 2023-10-18T16:07:37.845 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-18T16:07:37.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.954+0000 7fb3102df700 1 Processor -- start 2023-10-18T16:07:37.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.954+0000 7fb3102df700 1 -- start start 2023-10-18T16:07:37.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.955+0000 7fb3102df700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fb3080c38c0 0x7fb3080c3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:37.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.955+0000 7fb3102df700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb3080c0b70 0x7fb308149c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:37.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.955+0000 7fb3102df700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb30814d280 con 0x7fb30806a670 2023-10-18T16:07:37.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.955+0000 7fb3102df700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb308067520 con 0x7fb3080c0b70 2023-10-18T16:07:37.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.955+0000 7fb3102df700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb308151530 con 0x7fb3080c38c0 2023-10-18T16:07:37.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.955+0000 7fb30e87c700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb3080c0b70 0x7fb308149c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:37.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.955+0000 7fb30d87a700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fb30806a670 0x7fb30814d710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:52352/0 (socket says 172.21.15.88:52352) 2023-10-18T16:07:37.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.955+0000 7fb30e87c700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb3080c0b70 0x7fb308149c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43848/0 (socket says 172.21.15.88:43848) 2023-10-18T16:07:37.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.955+0000 7fb30d87a700 1 -- 172.21.15.88:0/679774793 learned_addr learned my addr 172.21.15.88:0/679774793 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:07:37.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.956+0000 7fb30e07b700 1 --2- 172.21.15.88:0/679774793 >> v2:172.21.15.151:3300/0 conn(0x7fb3080c38c0 0x7fb3080c3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:37.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.956+0000 7fb30e87c700 1 -- 172.21.15.88:0/679774793 >> v2:172.21.15.151:3300/0 conn(0x7fb3080c38c0 msgr2=0x7fb3080c3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:37.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.956+0000 7fb30e87c700 1 --2- 172.21.15.88:0/679774793 >> v2:172.21.15.151:3300/0 conn(0x7fb3080c38c0 0x7fb3080c3ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:37.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.956+0000 7fb30d079700 1 -- 172.21.15.88:0/679774793 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3149200572 0 0) 0x7fb30814d280 con 0x7fb30806a670 2023-10-18T16:07:37.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.956+0000 7fb30e87c700 1 -- 172.21.15.88:0/679774793 >> v1:172.21.15.88:6789/0 conn(0x7fb30806a670 legacy=0x7fb30814d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:37.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.956+0000 7fb30e87c700 1 -- 172.21.15.88:0/679774793 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3081509d0 con 0x7fb3080c0b70 2023-10-18T16:07:37.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.956+0000 7fb30e07b700 1 --2- 172.21.15.88:0/679774793 >> v2:172.21.15.151:3300/0 conn(0x7fb3080c38c0 0x7fb3080c3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-10-18T16:07:37.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.956+0000 7fb30e87c700 1 --2- 172.21.15.88:0/679774793 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb3080c0b70 0x7fb308149c30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fb308066820 tx=0x7fb300009060).ready entity=mon.0 client_cookie=48dfce57974d12a server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:37.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.956+0000 7fb30d079700 1 -- 172.21.15.88:0/679774793 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb30000b580 con 0x7fb3080c0b70 2023-10-18T16:07:37.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.956+0000 7fb30d079700 1 -- 172.21.15.88:0/679774793 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb30000b6e0 con 0x7fb3080c0b70 2023-10-18T16:07:37.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.956+0000 7fb30d079700 1 -- 172.21.15.88:0/679774793 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb30000ba60 con 0x7fb3080c0b70 2023-10-18T16:07:37.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.956+0000 7fb3102df700 1 -- 172.21.15.88:0/679774793 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb3080c0b70 msgr2=0x7fb308149c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:37.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.956+0000 7fb3102df700 1 --2- 172.21.15.88:0/679774793 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb3080c0b70 0x7fb308149c30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fb308066820 tx=0x7fb300009060).stop 2023-10-18T16:07:37.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.956+0000 7fb3102df700 1 -- 172.21.15.88:0/679774793 shutdown_connections 2023-10-18T16:07:37.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.956+0000 7fb3102df700 1 --2- 172.21.15.88:0/679774793 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb3080c0b70 0x7fb308149c30 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:37.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.956+0000 7fb3102df700 1 --2- 172.21.15.88:0/679774793 >> v2:172.21.15.151:3300/0 conn(0x7fb3080c38c0 0x7fb3080c3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:37.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.956+0000 7fb3102df700 1 -- 172.21.15.88:0/679774793 >> 172.21.15.88:0/679774793 conn(0x7fb308065cd0 msgr2=0x7fb3080660e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:37.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.956+0000 7fb3102df700 1 -- 172.21.15.88:0/679774793 shutdown_connections 2023-10-18T16:07:37.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.956+0000 7fb3102df700 1 -- 172.21.15.88:0/679774793 wait complete. 2023-10-18T16:07:37.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.957+0000 7fb3102df700 1 Processor -- start 2023-10-18T16:07:37.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.958+0000 7fb3102df700 1 -- start start 2023-10-18T16:07:37.969 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.958+0000 7fb3102df700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb30806a670 0x7fb3080c2050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:37.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.958+0000 7fb3102df700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb3080c0b70 0x7fb30814f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:37.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.958+0000 7fb3102df700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb3080c38c0 0x7fb3081600f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:37.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.958+0000 7fb3102df700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3080c1640 con 0x7fb30806a670 2023-10-18T16:07:37.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.958+0000 7fb3102df700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3080be360 con 0x7fb3080c0b70 2023-10-18T16:07:37.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.958+0000 7fb3102df700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3080c1bc0 con 0x7fb3080c38c0 2023-10-18T16:07:37.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.958+0000 7fb30e07b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb30806a670 0x7fb3080c2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:37.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.958+0000 7fb30e07b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb30806a670 0x7fb3080c2050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43854/0 (socket says 172.21.15.88:43854) 2023-10-18T16:07:37.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.958+0000 7fb30e07b700 1 -- 172.21.15.88:0/1994526540 learned_addr learned my addr 172.21.15.88:0/1994526540 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:37.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.959+0000 7fb30e07b700 1 -- 172.21.15.88:0/1994526540 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb3080c38c0 msgr2=0x7fb3081600f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:37.972 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.959+0000 7fb30e07b700 1 --2- 172.21.15.88:0/1994526540 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb3080c38c0 0x7fb3081600f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-10-18T16:07:37.972 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.959+0000 7fb30e07b700 1 -- 172.21.15.88:0/1994526540 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb3080c0b70 msgr2=0x7fb30814f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:37.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.959+0000 7fb30d87a700 1 --2- 172.21.15.88:0/1994526540 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb3080c0b70 0x7fb30814f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:37.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.959+0000 7fb30e07b700 1 --2- 172.21.15.88:0/1994526540 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb3080c0b70 0x7fb30814f9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:37.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.959+0000 7fb30e07b700 1 -- 172.21.15.88:0/1994526540 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3000092a0 con 0x7fb30806a670 2023-10-18T16:07:37.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.959+0000 7fb30e07b700 1 --2- 172.21.15.88:0/1994526540 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb30806a670 0x7fb3080c2050 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fb304007f90 tx=0x7fb304007870).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:37.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.959+0000 7fb2fe7fc700 1 -- 172.21.15.88:0/1994526540 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb304016400 con 0x7fb30806a670 2023-10-18T16:07:37.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.959+0000 7fb2fe7fc700 1 -- 172.21.15.88:0/1994526540 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb304016560 con 0x7fb30806a670 2023-10-18T16:07:37.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.959+0000 7fb3102df700 1 -- 172.21.15.88:0/1994526540 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3080be590 con 0x7fb30806a670 2023-10-18T16:07:37.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.959+0000 7fb2fe7fc700 1 -- 172.21.15.88:0/1994526540 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3040168e0 con 0x7fb30806a670 2023-10-18T16:07:37.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.959+0000 7fb3102df700 1 -- 172.21.15.88:0/1994526540 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3080c3350 con 0x7fb30806a670 2023-10-18T16:07:37.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.960+0000 7fb3102df700 1 -- 172.21.15.88:0/1994526540 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3080c3570 con 0x7fb30806a670 2023-10-18T16:07:37.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.960+0000 7fb2fe7fc700 1 -- 172.21.15.88:0/1994526540 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7fb30401f3f0 con 0x7fb30806a670 2023-10-18T16:07:37.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.960+0000 7fb2fe7fc700 1 --2- 172.21.15.88:0/1994526540 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fb2ec05f150 0x7fb2ec061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:37.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.961+0000 7fb2fe7fc700 1 -- 172.21.15.88:0/1994526540 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5269+0+0 (secure 0 0 0) 0x7fb30407a940 con 0x7fb30806a670 2023-10-18T16:07:37.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.961+0000 7fb30d87a700 1 --2- 172.21.15.88:0/1994526540 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fb2ec05f150 0x7fb2ec061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:37.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.961+0000 7fb30d87a700 1 --2- 172.21.15.88:0/1994526540 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fb2ec05f150 0x7fb2ec061610 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fb308151500 tx=0x7fb2f8009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:37.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:37.965+0000 7fb2fe7fc700 1 -- 172.21.15.88:0/1994526540 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7fb30404bf00 con 0x7fb30806a670 2023-10-18T16:07:38.119 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-18T16:07:38.122 DEBUG:teuthology.orchestra.run.smithi151:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-18T16:07:38.131 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.128+0000 7fb3102df700 1 -- 172.21.15.88:0/1994526540 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb30814cf20 con 0x7fb30806a670 2023-10-18T16:07:38.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.129+0000 7fb2fe7fc700 1 -- 172.21.15.88:0/1994526540 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+13709 (secure 0 0 0) 0x7fb30404f520 con 0x7fb30806a670 2023-10-18T16:07:38.133 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:07:38.133 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":18,"fsid":"fb1f77e8-ead0-4f90-8128-fdf5137eb16f","created":"2023-10-18T16:07:23.662468+0000","modified":"2023-10-18T16:07:37.534271+0000","last_up_change":"2023-10-18T16:07:32.498203+0000","last_in_change":"2023-10-18T16:07:26.064784+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-10-18T16:07:32.536320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-18T16:07:34.769989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"e421c20c-f7bc-4574-97a2-201bb4a16b4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":91007},{"type":"v1","addr":"172.21.15.88:6809","nonce":91007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":91007},{"type":"v1","addr":"172.21.15.88:6811","nonce":91007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":91007},{"type":"v1","addr":"172.21.15.88:6815","nonce":91007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":91007},{"type":"v1","addr":"172.21.15.88:6813","nonce":91007}]},"public_addr":"172.21.15.88:6809/91007","cluster_addr":"172.21.15.88:6811/91007","heartbeat_back_addr":"172.21.15.88:6815/91007","heartbeat_front_addr":"172.21.15.88:6813/91007","state":["exists","up"]},{"osd":1,"uuid":"9485c5ec-379a-4038-a2d8-1f1081fe2eea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":91009},{"type":"v1","addr":"172.21.15.88:6825","nonce":91009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":91009},{"type":"v1","addr":"172.21.15.88:6827","nonce":91009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":91009},{"type":"v1","addr":"172.21.15.88:6831","nonce":91009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":91009},{"type":"v1","addr":"172.21.15.88:6829","nonce":91009}]},"public_addr":"172.21.15.88:6825/91009","cluster_addr":"172.21.15.88:6827/91009","heartbeat_back_addr":"172.21.15.88:6831/91009","heartbeat_front_addr":"172.21.15.88:6829/91009","state":["exists","up"]},{"osd":2,"uuid":"671895e2-57f3-4f1a-ba4a-a580af1f3ec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":91005},{"type":"v1","addr":"172.21.15.88:6801","nonce":91005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":91005},{"type":"v1","addr":"172.21.15.88:6803","nonce":91005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":91005},{"type":"v1","addr":"172.21.15.88:6807","nonce":91005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":91005},{"type":"v1","addr":"172.21.15.88:6805","nonce":91005}]},"public_addr":"172.21.15.88:6801/91005","cluster_addr":"172.21.15.88:6803/91005","heartbeat_back_addr":"172.21.15.88:6807/91005","heartbeat_front_addr":"172.21.15.88:6805/91005","state":["exists","up"]},{"osd":3,"uuid":"0c4e6e7f-7d9c-4e65-beec-fd0ca384f320","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":91008},{"type":"v1","addr":"172.21.15.88:6817","nonce":91008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":91008},{"type":"v1","addr":"172.21.15.88:6819","nonce":91008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":91008},{"type":"v1","addr":"172.21.15.88:6823","nonce":91008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":91008},{"type":"v1","addr":"172.21.15.88:6821","nonce":91008}]},"public_addr":"172.21.15.88:6817/91008","cluster_addr":"172.21.15.88:6819/91008","heartbeat_back_addr":"172.21.15.88:6823/91008","heartbeat_front_addr":"172.21.15.88:6821/91008","state":["exists","up"]},{"osd":4,"uuid":"29570691-ef35-4b80-9247-aa3703877af4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":91514},{"type":"v1","addr":"172.21.15.151:6801","nonce":91514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":91514},{"type":"v1","addr":"172.21.15.151:6803","nonce":91514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":91514},{"type":"v1","addr":"172.21.15.151:6807","nonce":91514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":91514},{"type":"v1","addr":"172.21.15.151:6805","nonce":91514}]},"public_addr":"172.21.15.151:6801/91514","cluster_addr":"172.21.15.151:6803/91514","heartbeat_back_addr":"172.21.15.151:6807/91514","heartbeat_front_addr":"172.21.15.151:6805/91514","state":["exists","up"]},{"osd":5,"uuid":"26540b7b-032a-4b3b-bf6c-b131a449fa99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":91513},{"type":"v1","addr":"172.21.15.151:6809","nonce":91513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":91513},{"type":"v1","addr":"172.21.15.151:6811","nonce":91513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":91513},{"type":"v1","addr":"172.21.15.151:6815","nonce":91513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":91513},{"type":"v1","addr":"172.21.15.151:6813","nonce":91513}]},"public_addr":"172.21.15.151:6809/91513","cluster_addr":"172.21.15.151:6811/91513","heartbeat_back_addr":"172.21.15.151:6815/91513","heartbeat_front_addr":"172.21.15.151:6813/91513","state":["exists","up"]},{"osd":6,"uuid":"7af01f9c-90db-45fd-93c8-09517c9b0881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":91544},{"type":"v1","addr":"172.21.15.151:6817","nonce":91544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":91544},{"type":"v1","addr":"172.21.15.151:6819","nonce":91544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":91544},{"type":"v1","addr":"172.21.15.151:6823","nonce":91544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":91544},{"type":"v1","addr":"172.21.15.151:6821","nonce":91544}]},"public_addr":"172.21.15.151:6817/91544","cluster_addr":"172.21.15.151:6819/91544","heartbeat_back_addr":"172.21.15.151:6823/91544","heartbeat_front_addr":"172.21.15.151:6821/91544","state":["exists","up"]},{"osd":7,"uuid":"b527b0a5-4249-4854-a30f-bff897757239","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":91545},{"type":"v1","addr":"172.21.15.151:6825","nonce":91545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":91545},{"type":"v1","addr":"172.21.15.151:6827","nonce":91545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":91545},{"type":"v1","addr":"172.21.15.151:6831","nonce":91545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":91545},{"type":"v1","addr":"172.21.15.151:6829","nonce":91545}]},"public_addr":"172.21.15.151:6825/91545","cluster_addr":"172.21.15.151:6827/91545","heartbeat_back_addr":"172.21.15.151:6831/91545","heartbeat_front_addr":"172.21.15.151:6829/91545","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T16:07:31.216742+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T16:07:31.467229+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T16:07:31.220809+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T16:07:31.414999+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T16:07:31.188976+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T16:07:31.252153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T16:07:31.254324+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T16:07:31.203434+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-18T16:07:38.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.131+0000 7fb3102df700 1 -- 172.21.15.88:0/1994526540 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fb2ec05f150 msgr2=0x7fb2ec061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:38.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.131+0000 7fb3102df700 1 --2- 172.21.15.88:0/1994526540 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fb2ec05f150 0x7fb2ec061610 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fb308151500 tx=0x7fb2f8009040).stop 2023-10-18T16:07:38.136 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.131+0000 7fb3102df700 1 -- 172.21.15.88:0/1994526540 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb30806a670 msgr2=0x7fb3080c2050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:38.136 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.131+0000 7fb3102df700 1 --2- 172.21.15.88:0/1994526540 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb30806a670 0x7fb3080c2050 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fb304007f90 tx=0x7fb304007870).stop 2023-10-18T16:07:38.136 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.131+0000 7fb3102df700 1 -- 172.21.15.88:0/1994526540 shutdown_connections 2023-10-18T16:07:38.136 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.131+0000 7fb3102df700 1 --2- 172.21.15.88:0/1994526540 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fb2ec05f150 0x7fb2ec061610 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:38.136 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.131+0000 7fb3102df700 1 --2- 172.21.15.88:0/1994526540 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb3080c38c0 0x7fb3081600f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-10-18T16:07:38.136 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.131+0000 7fb3102df700 1 --2- 172.21.15.88:0/1994526540 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb30806a670 0x7fb3080c2050 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:38.137 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.131+0000 7fb3102df700 1 --2- 172.21.15.88:0/1994526540 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb3080c0b70 0x7fb30814f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:38.137 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.131+0000 7fb3102df700 1 -- 172.21.15.88:0/1994526540 >> 172.21.15.88:0/1994526540 conn(0x7fb308065cd0 msgr2=0x7fb3080bc840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:38.137 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.131+0000 7fb3102df700 1 -- 172.21.15.88:0/1994526540 shutdown_connections 2023-10-18T16:07:38.137 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.131+0000 7fb3102df700 1 -- 172.21.15.88:0/1994526540 wait complete. 2023-10-18T16:07:38.147 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2023-10-18T16:07:32.536320+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-10-18T16:07:34.769989+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2023-10-18T16:07:38.148 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2023-10-18T16:07:38.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.325+0000 7f5547b91700 1 Processor -- start 2023-10-18T16:07:38.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.325+0000 7f5547b91700 1 -- start start 2023-10-18T16:07:38.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.326+0000 7f5547b91700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f554014d9d0 0x7f554014ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:38.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.326+0000 7f5547b91700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5540068a50 0x7f5540062e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:38.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.326+0000 7f5547b91700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5540069ea0 con 0x7f55400befb0 2023-10-18T16:07:38.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.326+0000 7f5547b91700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f554014aca0 con 0x7f554014d9d0 2023-10-18T16:07:38.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.326+0000 7f5547b91700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5540068fb0 con 0x7f5540068a50 2023-10-18T16:07:38.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.326+0000 7f554512c700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5540068a50 0x7f5540062e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:38.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.326+0000 7f554592d700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f554014d9d0 0x7f554014ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:38.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.327+0000 7f554612e700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f55400befb0 0x7f55400bb420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.88:59946/0 (socket says 172.21.15.88:59946) 2023-10-18T16:07:38.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.327+0000 7f554612e700 1 -- 172.21.15.88:0/2020270981 learned_addr learned my addr 172.21.15.88:0/2020270981 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:07:38.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.327+0000 7f554592d700 1 -- 172.21.15.88:0/2020270981 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5540068a50 msgr2=0x7f5540062e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:38.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.327+0000 7f554592d700 1 --2- 172.21.15.88:0/2020270981 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5540068a50 0x7f5540062e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:38.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.327+0000 7f554592d700 1 -- 172.21.15.88:0/2020270981 >> v1:172.21.15.151:6789/0 conn(0x7f55400befb0 legacy=0x7f55400bb420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:38.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.327+0000 7f554592d700 1 -- 172.21.15.88:0/2020270981 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5540069290 con 0x7f554014d9d0 2023-10-18T16:07:38.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.327+0000 7f554512c700 1 --2- 172.21.15.88:0/2020270981 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5540068a50 0x7f5540062e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2023-10-18T16:07:38.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.327+0000 7f554592d700 1 --2- 172.21.15.88:0/2020270981 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f554014d9d0 0x7f554014ddb0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f553c009760 tx=0x7f553c009060).ready entity=mon.1 client_cookie=95c0e88e80c689d8 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:38.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.327+0000 7f554492b700 1 -- 172.21.15.88:0/2020270981 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f553c00b590 con 0x7f554014d9d0 2023-10-18T16:07:38.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.327+0000 7f554492b700 1 -- 172.21.15.88:0/2020270981 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f553c00b6f0 con 0x7f554014d9d0 2023-10-18T16:07:38.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.327+0000 7f554492b700 1 -- 172.21.15.88:0/2020270981 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f553c00ba70 con 0x7f554014d9d0 2023-10-18T16:07:38.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.328+0000 7f5547b91700 1 -- 172.21.15.88:0/2020270981 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f554014d9d0 msgr2=0x7f554014ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:38.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.328+0000 7f5547b91700 1 --2- 172.21.15.88:0/2020270981 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f554014d9d0 0x7f554014ddb0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f553c009760 tx=0x7f553c009060).stop 2023-10-18T16:07:38.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.328+0000 7f5547b91700 1 -- 172.21.15.88:0/2020270981 shutdown_connections 2023-10-18T16:07:38.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.328+0000 7f5547b91700 1 --2- 172.21.15.88:0/2020270981 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f554014d9d0 0x7f554014ddb0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:38.338 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.328+0000 7f5547b91700 1 --2- 172.21.15.88:0/2020270981 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5540068a50 0x7f5540062e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:38.338 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.328+0000 7f5547b91700 1 -- 172.21.15.88:0/2020270981 >> 172.21.15.88:0/2020270981 conn(0x7f554005e050 msgr2=0x7f5540062650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:38.338 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.328+0000 7f5547b91700 1 -- 172.21.15.88:0/2020270981 shutdown_connections 2023-10-18T16:07:38.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.328+0000 7f5547b91700 1 -- 172.21.15.88:0/2020270981 wait complete. 2023-10-18T16:07:38.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.329+0000 7f5547b91700 1 Processor -- start 2023-10-18T16:07:38.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.329+0000 7f5547b91700 1 -- start start 2023-10-18T16:07:38.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.330+0000 7f5547b91700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5540068a50 0x7f55400c1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:38.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.330+0000 7f5547b91700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f55400befb0 0x7f554014f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:38.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.330+0000 7f5547b91700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f554014d9d0 0x7f554014f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:38.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.330+0000 7f5547b91700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5540150a00 con 0x7f554014d9d0 2023-10-18T16:07:38.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.330+0000 7f5547b91700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f554014ec70 con 0x7f5540068a50 2023-10-18T16:07:38.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.330+0000 7f5547b91700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f554014fae0 con 0x7f55400befb0 2023-10-18T16:07:38.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.330+0000 7f554612e700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f554014d9d0 0x7f554014f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:38.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.330+0000 7f554592d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5540068a50 0x7f55400c1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:38.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.330+0000 7f554612e700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f554014d9d0 0x7f554014f5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43856/0 (socket says 172.21.15.88:43856) 2023-10-18T16:07:38.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.330+0000 7f554612e700 1 -- 172.21.15.88:0/2546273555 learned_addr learned my addr 172.21.15.88:0/2546273555 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:38.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.330+0000 7f554512c700 1 --2- 172.21.15.88:0/2546273555 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f55400befb0 0x7f554014f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:38.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.330+0000 7f554612e700 1 -- 172.21.15.88:0/2546273555 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f55400befb0 msgr2=0x7f554014f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:38.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.330+0000 7f554612e700 1 --2- 172.21.15.88:0/2546273555 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f55400befb0 0x7f554014f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:38.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.330+0000 7f554612e700 1 -- 172.21.15.88:0/2546273555 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5540068a50 msgr2=0x7f55400c1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:38.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.330+0000 7f554612e700 1 --2- 172.21.15.88:0/2546273555 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5540068a50 0x7f55400c1ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:38.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.330+0000 7f554612e700 1 -- 172.21.15.88:0/2546273555 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55300092a0 con 0x7f554014d9d0 2023-10-18T16:07:38.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.330+0000 7f554612e700 1 --2- 172.21.15.88:0/2546273555 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f554014d9d0 0x7f554014f5f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f553800c7c0 tx=0x7f55380015e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:38.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.330+0000 7f5535ffb700 1 -- 172.21.15.88:0/2546273555 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5538005d30 con 0x7f554014d9d0 2023-10-18T16:07:38.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.331+0000 7f5535ffb700 1 -- 172.21.15.88:0/2546273555 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5538005e90 con 0x7f554014d9d0 2023-10-18T16:07:38.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.331+0000 7f5547b91700 1 -- 172.21.15.88:0/2546273555 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f553c0092a0 con 0x7f554014d9d0 2023-10-18T16:07:38.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.331+0000 7f5535ffb700 1 -- 172.21.15.88:0/2546273555 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55380072e0 con 0x7f554014d9d0 2023-10-18T16:07:38.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.331+0000 7f5547b91700 1 -- 172.21.15.88:0/2546273555 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5540150c50 con 0x7f554014d9d0 2023-10-18T16:07:38.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.332+0000 7f5547b91700 1 -- 172.21.15.88:0/2546273555 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f554014ee00 con 0x7f554014d9d0 2023-10-18T16:07:38.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.332+0000 7f5535ffb700 1 -- 172.21.15.88:0/2546273555 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f55380076d0 con 0x7f554014d9d0 2023-10-18T16:07:38.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.332+0000 7f5535ffb700 1 --2- 172.21.15.88:0/2546273555 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f552405f180 0x7f5524061640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:38.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.332+0000 7f5535ffb700 1 -- 172.21.15.88:0/2546273555 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5269+0+0 (secure 0 0 0) 0x7f55380797b0 con 0x7f554014d9d0 2023-10-18T16:07:38.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.333+0000 7f554592d700 1 --2- 172.21.15.88:0/2546273555 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f552405f180 0x7f5524061640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:38.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.333+0000 7f554592d700 1 --2- 172.21.15.88:0/2546273555 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f552405f180 0x7f5524061640 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f553c00b350 tx=0x7f553c017000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:38.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.336+0000 7f5535ffb700 1 -- 172.21.15.88:0/2546273555 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f553804add0 con 0x7f554014d9d0 2023-10-18T16:07:38.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.501+0000 7f5547b91700 1 -- 172.21.15.88:0/2546273555 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f554006c4e0 con 0x7f554014d9d0 2023-10-18T16:07:38.505 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.501+0000 7f5535ffb700 1 -- 172.21.15.88:0/2546273555 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v18) v1 ==== 110+0+10 (secure 0 0 0) 0x7f553804e3f0 con 0x7f554014d9d0 2023-10-18T16:07:38.505 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 1 2023-10-18T16:07:38.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.503+0000 7f5547b91700 1 -- 172.21.15.88:0/2546273555 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f552405f180 msgr2=0x7f5524061640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:38.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.503+0000 7f5547b91700 1 --2- 172.21.15.88:0/2546273555 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f552405f180 0x7f5524061640 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f553c00b350 tx=0x7f553c017000).stop 2023-10-18T16:07:38.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.503+0000 7f5547b91700 1 -- 172.21.15.88:0/2546273555 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f554014d9d0 msgr2=0x7f554014f5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:38.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.503+0000 7f5547b91700 1 --2- 172.21.15.88:0/2546273555 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f554014d9d0 0x7f554014f5f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f553800c7c0 tx=0x7f55380015e0).stop 2023-10-18T16:07:38.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.503+0000 7f5547b91700 1 -- 172.21.15.88:0/2546273555 shutdown_connections 2023-10-18T16:07:38.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.503+0000 7f5547b91700 1 --2- 172.21.15.88:0/2546273555 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f552405f180 0x7f5524061640 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:38.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.503+0000 7f5547b91700 1 --2- 172.21.15.88:0/2546273555 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f554014d9d0 0x7f554014f5f0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:38.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.503+0000 7f5547b91700 1 --2- 172.21.15.88:0/2546273555 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5540068a50 0x7f55400c1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:38.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.503+0000 7f5547b91700 1 --2- 172.21.15.88:0/2546273555 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f55400befb0 0x7f554014f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:38.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.503+0000 7f5547b91700 1 -- 172.21.15.88:0/2546273555 >> 172.21.15.88:0/2546273555 conn(0x7f554005e050 msgr2=0x7f5540148dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:38.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.503+0000 7f5547b91700 1 -- 172.21.15.88:0/2546273555 shutdown_connections 2023-10-18T16:07:38.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.503+0000 7f5547b91700 1 -- 172.21.15.88:0/2546273555 wait complete. 2023-10-18T16:07:38.518 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-10-18T16:07:38.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.632+0000 7f6d104ee700 1 Processor -- start 2023-10-18T16:07:38.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.632+0000 7f6d104ee700 1 -- start start 2023-10-18T16:07:38.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.633+0000 7f6d104ee700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6d081362d0 0x7f6d08150360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:38.640 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.633+0000 7f6d104ee700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f6d08069b80 0x7f6d080c0170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:38.640 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.633+0000 7f6d104ee700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d08153ce0 con 0x7f6d081366b0 2023-10-18T16:07:38.640 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.633+0000 7f6d104ee700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d080c0fd0 con 0x7f6d08069b80 2023-10-18T16:07:38.640 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.633+0000 7f6d104ee700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d08150910 con 0x7f6d081362d0 2023-10-18T16:07:38.641 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.633+0000 7f6d0da89700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f6d081366b0 0x7f6d08154170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:52356/0 (socket says 172.21.15.88:52356) 2023-10-18T16:07:38.641 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.633+0000 7f6d0da89700 1 -- 172.21.15.88:0/3202720135 learned_addr learned my addr 172.21.15.88:0/3202720135 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:07:38.641 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.634+0000 7f6d0e28a700 1 --2- 172.21.15.88:0/3202720135 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6d081362d0 0x7f6d08150360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:38.641 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.634+0000 7f6d0ea8b700 1 --2- 172.21.15.88:0/3202720135 >> v2:172.21.15.151:3300/0 conn(0x7f6d08069b80 0x7f6d080c0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:38.641 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.634+0000 7f6d0d288700 1 -- 172.21.15.88:0/3202720135 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4045750761 0 0) 0x7f6d08153ce0 con 0x7f6d081366b0 2023-10-18T16:07:38.642 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.634+0000 7f6d0d288700 1 -- 172.21.15.88:0/3202720135 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cf40033f0 con 0x7f6d081366b0 2023-10-18T16:07:38.642 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.634+0000 7f6d0d288700 1 -- 172.21.15.88:0/3202720135 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3908400827 0 0) 0x7f6cf40033f0 con 0x7f6d081366b0 2023-10-18T16:07:38.642 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.634+0000 7f6d0d288700 1 -- 172.21.15.88:0/3202720135 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6d081362d0 msgr2=0x7f6d08150360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:38.642 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.634+0000 7f6d0d288700 1 --2- 172.21.15.88:0/3202720135 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6d081362d0 0x7f6d08150360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:38.642 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.634+0000 7f6d0d288700 1 -- 172.21.15.88:0/3202720135 >> v2:172.21.15.151:3300/0 conn(0x7f6d08069b80 msgr2=0x7f6d080c0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:38.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.634+0000 7f6d0d288700 1 --2- 172.21.15.88:0/3202720135 >> v2:172.21.15.151:3300/0 conn(0x7f6d08069b80 0x7f6d080c0170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:38.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.634+0000 7f6d0d288700 1 -- 172.21.15.88:0/3202720135 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d08150bf0 con 0x7f6d081366b0 2023-10-18T16:07:38.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.634+0000 7f6d0d288700 1 -- 172.21.15.88:0/3202720135 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441715075 0 0) 0x7f6cf8001470 con 0x7f6d081366b0 2023-10-18T16:07:38.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.634+0000 7f6d0d288700 1 -- 172.21.15.88:0/3202720135 >> v1:172.21.15.88:6789/0 conn(0x7f6d081366b0 legacy=0x7f6d08154170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:38.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.634+0000 7f6d0d288700 1 --2- 172.21.15.88:0/3202720135 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6cf40038e0 0x7f6cf4003d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:38.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.634+0000 7f6d0d288700 1 --2- 172.21.15.88:0/3202720135 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6d081362d0 0x7f6cf4004690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:38.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.634+0000 7f6d0d288700 1 --2- 172.21.15.88:0/3202720135 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6cf4004b80 0x7f6cf4005000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:38.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.634+0000 7f6d0d288700 1 -- 172.21.15.88:0/3202720135 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d080c0fd0 con 0x7f6cf40038e0 2023-10-18T16:07:38.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.634+0000 7f6d0d288700 1 -- 172.21.15.88:0/3202720135 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d08150910 con 0x7f6cf4004b80 2023-10-18T16:07:38.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.634+0000 7f6d0d288700 1 -- 172.21.15.88:0/3202720135 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf400b8e0 con 0x7f6d081362d0 2023-10-18T16:07:38.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.634+0000 7f6d0e28a700 1 --2- 172.21.15.88:0/3202720135 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6cf40038e0 0x7f6cf4003d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:38.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.634+0000 7f6d0ea8b700 1 --2- 172.21.15.88:0/3202720135 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6cf4004b80 0x7f6cf4005000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:38.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.634+0000 7f6d0da89700 1 --2- 172.21.15.88:0/3202720135 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6d081362d0 0x7f6cf4004690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:38.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.634+0000 7f6d0e28a700 1 -- 172.21.15.88:0/3202720135 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6d081362d0 msgr2=0x7f6cf4004690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:38.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.634+0000 7f6d0e28a700 1 --2- 172.21.15.88:0/3202720135 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6d081362d0 0x7f6cf4004690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:38.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.634+0000 7f6d0e28a700 1 -- 172.21.15.88:0/3202720135 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6cf4004b80 msgr2=0x7f6cf4005000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:38.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.634+0000 7f6d0e28a700 1 --2- 172.21.15.88:0/3202720135 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6cf4004b80 0x7f6cf4005000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:38.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.634+0000 7f6d0e28a700 1 -- 172.21.15.88:0/3202720135 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cf40033f0 con 0x7f6cf40038e0 2023-10-18T16:07:38.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.634+0000 7f6d0da89700 1 --2- 172.21.15.88:0/3202720135 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6d081362d0 0x7f6cf4004690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:07:38.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.634+0000 7f6d0e28a700 1 --2- 172.21.15.88:0/3202720135 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6cf40038e0 0x7f6cf4003d00 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f6d0400e210 tx=0x7f6d0400b040).ready entity=mon.0 client_cookie=5cd28a351196b9c7 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:38.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.635+0000 7f6d0d288700 1 -- 172.21.15.88:0/3202720135 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d04009ac0 con 0x7f6cf40038e0 2023-10-18T16:07:38.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.635+0000 7f6d0d288700 1 -- 172.21.15.88:0/3202720135 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d0400f040 con 0x7f6cf40038e0 2023-10-18T16:07:38.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.635+0000 7f6d0d288700 1 -- 172.21.15.88:0/3202720135 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d0400ecd0 con 0x7f6cf40038e0 2023-10-18T16:07:38.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.635+0000 7f6d104ee700 1 -- 172.21.15.88:0/3202720135 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6cf40038e0 msgr2=0x7f6cf4003d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:38.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.635+0000 7f6d104ee700 1 --2- 172.21.15.88:0/3202720135 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6cf40038e0 0x7f6cf4003d00 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f6d0400e210 tx=0x7f6d0400b040).stop 2023-10-18T16:07:38.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.635+0000 7f6d0ea8b700 1 -- 172.21.15.88:0/3202720135 reap_dead start 2023-10-18T16:07:38.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.635+0000 7f6d104ee700 1 -- 172.21.15.88:0/3202720135 shutdown_connections 2023-10-18T16:07:38.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.635+0000 7f6d104ee700 1 -- 172.21.15.88:0/3202720135 >> 172.21.15.88:0/3202720135 conn(0x7f6d08063760 msgr2=0x7f6d08063b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:38.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.635+0000 7f6d104ee700 1 -- 172.21.15.88:0/3202720135 shutdown_connections 2023-10-18T16:07:38.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.635+0000 7f6d104ee700 1 -- 172.21.15.88:0/3202720135 wait complete. 2023-10-18T16:07:38.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.636+0000 7f6d104ee700 1 Processor -- start 2023-10-18T16:07:38.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.636+0000 7f6d104ee700 1 -- start start 2023-10-18T16:07:38.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.637+0000 7f6d104ee700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6cf40038e0 0x7f6d080c27a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:38.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.637+0000 7f6d104ee700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6d08160000 0x7f6d080c3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:38.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.637+0000 7f6d104ee700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6d080c2c90 0x7f6d080c3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:38.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.637+0000 7f6d104ee700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d080c1700 con 0x7f6cf40038e0 2023-10-18T16:07:38.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.637+0000 7f6d104ee700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d080c19e0 con 0x7f6d080c2c90 2023-10-18T16:07:38.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.637+0000 7f6d104ee700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d080c1cc0 con 0x7f6d08160000 2023-10-18T16:07:38.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.637+0000 7f6d0e28a700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6cf40038e0 0x7f6d080c27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:38.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.637+0000 7f6d0e28a700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6cf40038e0 0x7f6d080c27a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43872/0 (socket says 172.21.15.88:43872) 2023-10-18T16:07:38.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.637+0000 7f6d0e28a700 1 -- 172.21.15.88:0/1152707156 learned_addr learned my addr 172.21.15.88:0/1152707156 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:38.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.637+0000 7f6d0da89700 1 --2- 172.21.15.88:0/1152707156 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6d08160000 0x7f6d080c3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:38.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.637+0000 7f6d0ea8b700 1 --2- 172.21.15.88:0/1152707156 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6d080c2c90 0x7f6d080c3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:38.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.637+0000 7f6d0e28a700 1 -- 172.21.15.88:0/1152707156 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6d08160000 msgr2=0x7f6d080c3ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:38.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.637+0000 7f6d0e28a700 1 --2- 172.21.15.88:0/1152707156 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6d08160000 0x7f6d080c3ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:38.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.637+0000 7f6d0e28a700 1 -- 172.21.15.88:0/1152707156 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6d080c2c90 msgr2=0x7f6d080c3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:38.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.637+0000 7f6d0e28a700 1 --2- 172.21.15.88:0/1152707156 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6d080c2c90 0x7f6d080c3fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:38.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.637+0000 7f6d0e28a700 1 -- 172.21.15.88:0/1152707156 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d080c1fa0 con 0x7f6cf40038e0 2023-10-18T16:07:38.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.637+0000 7f6d0ea8b700 1 --2- 172.21.15.88:0/1152707156 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6d080c2c90 0x7f6d080c3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:07:38.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.637+0000 7f6d0e28a700 1 --2- 172.21.15.88:0/1152707156 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6cf40038e0 0x7f6d080c27a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f6cf4004490 tx=0x7f6d0401a000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:38.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.638+0000 7f6cfe7fc700 1 -- 172.21.15.88:0/1152707156 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d0401dcc0 con 0x7f6cf40038e0 2023-10-18T16:07:38.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.638+0000 7f6cfe7fc700 1 -- 172.21.15.88:0/1152707156 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d0401de20 con 0x7f6cf40038e0 2023-10-18T16:07:38.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.638+0000 7f6d104ee700 1 -- 172.21.15.88:0/1152707156 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d080c2280 con 0x7f6cf40038e0 2023-10-18T16:07:38.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.638+0000 7f6cfe7fc700 1 -- 172.21.15.88:0/1152707156 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d04013650 con 0x7f6cf40038e0 2023-10-18T16:07:38.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.638+0000 7f6d104ee700 1 -- 172.21.15.88:0/1152707156 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d080b8300 con 0x7f6cf40038e0 2023-10-18T16:07:38.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.639+0000 7f6d104ee700 1 -- 172.21.15.88:0/1152707156 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d0806c4e0 con 0x7f6cf40038e0 2023-10-18T16:07:38.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.639+0000 7f6cfe7fc700 1 -- 172.21.15.88:0/1152707156 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f6d04013a80 con 0x7f6cf40038e0 2023-10-18T16:07:38.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.639+0000 7f6cfe7fc700 1 --2- 172.21.15.88:0/1152707156 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f6cec05f150 0x7f6cec061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:38.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.639+0000 7f6cfe7fc700 1 -- 172.21.15.88:0/1152707156 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5269+0+0 (secure 0 0 0) 0x7f6d0407b850 con 0x7f6cf40038e0 2023-10-18T16:07:38.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.639+0000 7f6d0da89700 1 --2- 172.21.15.88:0/1152707156 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f6cec05f150 0x7f6cec061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:38.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.640+0000 7f6d0da89700 1 --2- 172.21.15.88:0/1152707156 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f6cec05f150 0x7f6cec061610 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f6cf80010e0 tx=0x7f6cf8007040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:38.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.643+0000 7f6cfe7fc700 1 -- 172.21.15.88:0/1152707156 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f6d0404cf60 con 0x7f6cf40038e0 2023-10-18T16:07:38.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.806+0000 7f6d104ee700 1 -- 172.21.15.88:0/1152707156 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f6d0806a240 con 0x7f6cf40038e0 2023-10-18T16:07:38.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.806+0000 7f6cfe7fc700 1 -- 172.21.15.88:0/1152707156 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v18) v1 ==== 92+0+10 (secure 0 0 0) 0x7f6d0402a030 con 0x7f6cf40038e0 2023-10-18T16:07:38.810 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 8 2023-10-18T16:07:38.812 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.808+0000 7f6d104ee700 1 -- 172.21.15.88:0/1152707156 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f6cec05f150 msgr2=0x7f6cec061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:38.812 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.808+0000 7f6d104ee700 1 --2- 172.21.15.88:0/1152707156 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f6cec05f150 0x7f6cec061610 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f6cf80010e0 tx=0x7f6cf8007040).stop 2023-10-18T16:07:38.812 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.808+0000 7f6d104ee700 1 -- 172.21.15.88:0/1152707156 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6cf40038e0 msgr2=0x7f6d080c27a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:38.813 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.808+0000 7f6d104ee700 1 --2- 172.21.15.88:0/1152707156 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6cf40038e0 0x7f6d080c27a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f6cf4004490 tx=0x7f6d0401a000).stop 2023-10-18T16:07:38.813 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.808+0000 7f6d104ee700 1 -- 172.21.15.88:0/1152707156 shutdown_connections 2023-10-18T16:07:38.813 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.808+0000 7f6d104ee700 1 --2- 172.21.15.88:0/1152707156 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f6cec05f150 0x7f6cec061610 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:38.813 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.808+0000 7f6d104ee700 1 --2- 172.21.15.88:0/1152707156 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6d080c2c90 0x7f6d080c3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:38.813 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.808+0000 7f6d104ee700 1 --2- 172.21.15.88:0/1152707156 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6cf40038e0 0x7f6d080c27a0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:38.814 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.808+0000 7f6d104ee700 1 --2- 172.21.15.88:0/1152707156 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6d08160000 0x7f6d080c3ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:38.814 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.808+0000 7f6d104ee700 1 -- 172.21.15.88:0/1152707156 >> 172.21.15.88:0/1152707156 conn(0x7f6d08063760 msgr2=0x7f6d0814e1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:38.814 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.808+0000 7f6d104ee700 1 -- 172.21.15.88:0/1152707156 shutdown_connections 2023-10-18T16:07:38.814 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.808+0000 7f6d104ee700 1 -- 172.21.15.88:0/1152707156 wait complete. 2023-10-18T16:07:38.823 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-18T16:07:38.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.932+0000 7f70774ff700 1 Processor -- start 2023-10-18T16:07:38.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.932+0000 7f70774ff700 1 -- start start 2023-10-18T16:07:38.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.933+0000 7f70774ff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f70700c38c0 0x7f70700c3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:38.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.933+0000 7f70774ff700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f70700c0b70 0x7f7070149c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:38.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.933+0000 7f70774ff700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f707014d280 con 0x7f707006a670 2023-10-18T16:07:38.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.933+0000 7f70774ff700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7070067520 con 0x7f70700c38c0 2023-10-18T16:07:38.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.933+0000 7f70774ff700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7070151530 con 0x7f70700c0b70 2023-10-18T16:07:38.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.933+0000 7f707529b700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f70700c38c0 0x7f70700c3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:38.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.933+0000 7f707529b700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f70700c38c0 0x7f70700c3ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43880/0 (socket says 172.21.15.88:43880) 2023-10-18T16:07:38.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.933+0000 7f707529b700 1 -- 172.21.15.88:0/3806276835 learned_addr learned my addr 172.21.15.88:0/3806276835 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:38.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.934+0000 7f7075a9c700 1 --2- 172.21.15.88:0/3806276835 >> v2:172.21.15.151:3300/0 conn(0x7f70700c0b70 0x7f7070149c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:38.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.934+0000 7f707529b700 1 -- 172.21.15.88:0/3806276835 >> v2:172.21.15.151:3300/0 conn(0x7f70700c0b70 msgr2=0x7f7070149c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:38.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.934+0000 7f707529b700 1 --2- 172.21.15.88:0/3806276835 >> v2:172.21.15.151:3300/0 conn(0x7f70700c0b70 0x7f7070149c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:38.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.934+0000 7f707529b700 1 -- 172.21.15.88:0/3806276835 >> v1:172.21.15.151:6789/0 conn(0x7f707006a670 legacy=0x7f707014d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:38.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.934+0000 7f707529b700 1 -- 172.21.15.88:0/3806276835 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70701509d0 con 0x7f70700c38c0 2023-10-18T16:07:38.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.934+0000 7f7075a9c700 1 --2- 172.21.15.88:0/3806276835 >> v2:172.21.15.151:3300/0 conn(0x7f70700c0b70 0x7f7070149c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-10-18T16:07:38.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.934+0000 7f707529b700 1 --2- 172.21.15.88:0/3806276835 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70700c38c0 0x7f70700c3ca0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f70700c1640 tx=0x7f706c009840).ready entity=mon.0 client_cookie=802e8b6eded02877 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:38.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.934+0000 7f7067fff700 1 -- 172.21.15.88:0/3806276835 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f706c00e070 con 0x7f70700c38c0 2023-10-18T16:07:38.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.934+0000 7f7067fff700 1 -- 172.21.15.88:0/3806276835 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f706c00ba10 con 0x7f70700c38c0 2023-10-18T16:07:38.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.934+0000 7f7067fff700 1 -- 172.21.15.88:0/3806276835 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f706c00bd90 con 0x7f70700c38c0 2023-10-18T16:07:38.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.934+0000 7f70774ff700 1 -- 172.21.15.88:0/3806276835 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70700c38c0 msgr2=0x7f70700c3ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:38.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.934+0000 7f70774ff700 1 --2- 172.21.15.88:0/3806276835 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70700c38c0 0x7f70700c3ca0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f70700c1640 tx=0x7f706c009840).stop 2023-10-18T16:07:38.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.934+0000 7f70774ff700 1 -- 172.21.15.88:0/3806276835 shutdown_connections 2023-10-18T16:07:38.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.934+0000 7f70774ff700 1 --2- 172.21.15.88:0/3806276835 >> v2:172.21.15.151:3300/0 conn(0x7f70700c0b70 0x7f7070149c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:38.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.934+0000 7f70774ff700 1 --2- 172.21.15.88:0/3806276835 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70700c38c0 0x7f70700c3ca0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:38.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.934+0000 7f70774ff700 1 -- 172.21.15.88:0/3806276835 >> 172.21.15.88:0/3806276835 conn(0x7f7070065cd0 msgr2=0x7f70700660e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:38.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.935+0000 7f70774ff700 1 -- 172.21.15.88:0/3806276835 shutdown_connections 2023-10-18T16:07:38.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.935+0000 7f70774ff700 1 -- 172.21.15.88:0/3806276835 wait complete. 2023-10-18T16:07:38.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.936+0000 7f70774ff700 1 Processor -- start 2023-10-18T16:07:38.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.936+0000 7f70774ff700 1 -- start start 2023-10-18T16:07:38.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.936+0000 7f70774ff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f707006a670 0x7f707014ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:38.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.936+0000 7f70774ff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f70700c0b70 0x7f707014ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:38.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.936+0000 7f70774ff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f70700c38c0 0x7f70701521b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:38.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.936+0000 7f70774ff700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f707014b9d0 con 0x7f707006a670 2023-10-18T16:07:38.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.936+0000 7f70774ff700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7070151fd0 con 0x7f70700c38c0 2023-10-18T16:07:38.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.936+0000 7f70774ff700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f707014c3b0 con 0x7f70700c0b70 2023-10-18T16:07:38.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.937+0000 7f707529b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f707006a670 0x7f707014ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:38.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.937+0000 7f707529b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f707006a670 0x7f707014ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43882/0 (socket says 172.21.15.88:43882) 2023-10-18T16:07:38.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.937+0000 7f707529b700 1 -- 172.21.15.88:0/3640396706 learned_addr learned my addr 172.21.15.88:0/3640396706 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:38.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.937+0000 7f7075a9c700 1 --2- 172.21.15.88:0/3640396706 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f70700c38c0 0x7f70701521b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:38.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.937+0000 7f7074a9a700 1 --2- 172.21.15.88:0/3640396706 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f70700c0b70 0x7f707014ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:38.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.937+0000 7f707529b700 1 -- 172.21.15.88:0/3640396706 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f70700c0b70 msgr2=0x7f707014ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:38.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.937+0000 7f707529b700 1 --2- 172.21.15.88:0/3640396706 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f70700c0b70 0x7f707014ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:38.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.937+0000 7f707529b700 1 -- 172.21.15.88:0/3640396706 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f70700c38c0 msgr2=0x7f70701521b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:38.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.937+0000 7f707529b700 1 --2- 172.21.15.88:0/3640396706 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f70700c38c0 0x7f70701521b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:38.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.937+0000 7f707529b700 1 -- 172.21.15.88:0/3640396706 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f706c009ae0 con 0x7f707006a670 2023-10-18T16:07:38.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.937+0000 7f707529b700 1 --2- 172.21.15.88:0/3640396706 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f707006a670 0x7f707014ae40 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f70700ba810 tx=0x7f706c009840).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:38.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.937+0000 7f70657fa700 1 -- 172.21.15.88:0/3640396706 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f706c013cf0 con 0x7f707006a670 2023-10-18T16:07:38.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.937+0000 7f70657fa700 1 -- 172.21.15.88:0/3640396706 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f706c013e50 con 0x7f707006a670 2023-10-18T16:07:38.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.937+0000 7f70774ff700 1 -- 172.21.15.88:0/3640396706 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70701534f0 con 0x7f707006a670 2023-10-18T16:07:38.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.937+0000 7f70657fa700 1 -- 172.21.15.88:0/3640396706 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f706c01c630 con 0x7f707006a670 2023-10-18T16:07:38.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.937+0000 7f70774ff700 1 -- 172.21.15.88:0/3640396706 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7070151cf0 con 0x7f707006a670 2023-10-18T16:07:38.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.938+0000 7f70774ff700 1 -- 172.21.15.88:0/3640396706 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70701537d0 con 0x7f707006a670 2023-10-18T16:07:38.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.938+0000 7f70657fa700 1 -- 172.21.15.88:0/3640396706 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f706c01c9a0 con 0x7f707006a670 2023-10-18T16:07:38.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.939+0000 7f70657fa700 1 --2- 172.21.15.88:0/3640396706 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f705405f120 0x7f70540615e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:38.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.939+0000 7f70657fa700 1 -- 172.21.15.88:0/3640396706 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5269+0+0 (secure 0 0 0) 0x7f706c07b870 con 0x7f707006a670 2023-10-18T16:07:38.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.939+0000 7f7074a9a700 1 --2- 172.21.15.88:0/3640396706 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f705405f120 0x7f70540615e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:38.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.939+0000 7f7074a9a700 1 --2- 172.21.15.88:0/3640396706 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f705405f120 0x7f70540615e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f707014a540 tx=0x7f7060008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:38.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:38.943+0000 7f70657fa700 1 -- 172.21.15.88:0/3640396706 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f706c04ce90 con 0x7f707006a670 2023-10-18T16:07:39.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.109+0000 7f70774ff700 1 -- 172.21.15.88:0/3640396706 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f70701537d0 con 0x7f707006a670 2023-10-18T16:07:39.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.109+0000 7f70657fa700 1 -- 172.21.15.88:0/3640396706 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v18) v1 ==== 84+0+143 (secure 0 0 0) 0x7f706c0504b0 con 0x7f707006a670 2023-10-18T16:07:39.113 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:07:39.114 INFO:teuthology.orchestra.run.smithi088.stdout:[ 2023-10-18T16:07:39.114 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-10-18T16:07:39.114 INFO:teuthology.orchestra.run.smithi088.stdout: "poolnum": 1, 2023-10-18T16:07:39.114 INFO:teuthology.orchestra.run.smithi088.stdout: "poolname": "device_health_metrics" 2023-10-18T16:07:39.115 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-10-18T16:07:39.115 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-10-18T16:07:39.115 INFO:teuthology.orchestra.run.smithi088.stdout: "poolnum": 2, 2023-10-18T16:07:39.115 INFO:teuthology.orchestra.run.smithi088.stdout: "poolname": "rbd" 2023-10-18T16:07:39.116 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-10-18T16:07:39.116 INFO:teuthology.orchestra.run.smithi088.stdout:] 2023-10-18T16:07:39.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.111+0000 7f70774ff700 1 -- 172.21.15.88:0/3640396706 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f705405f120 msgr2=0x7f70540615e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:39.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.111+0000 7f70774ff700 1 --2- 172.21.15.88:0/3640396706 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f705405f120 0x7f70540615e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f707014a540 tx=0x7f7060008040).stop 2023-10-18T16:07:39.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.111+0000 7f70774ff700 1 -- 172.21.15.88:0/3640396706 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f707006a670 msgr2=0x7f707014ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:39.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.111+0000 7f70774ff700 1 --2- 172.21.15.88:0/3640396706 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f707006a670 0x7f707014ae40 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f70700ba810 tx=0x7f706c009840).stop 2023-10-18T16:07:39.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.111+0000 7f70774ff700 1 -- 172.21.15.88:0/3640396706 shutdown_connections 2023-10-18T16:07:39.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.111+0000 7f70774ff700 1 --2- 172.21.15.88:0/3640396706 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f705405f120 0x7f70540615e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:39.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.111+0000 7f70774ff700 1 --2- 172.21.15.88:0/3640396706 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f70700c38c0 0x7f70701521b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:39.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.111+0000 7f70774ff700 1 --2- 172.21.15.88:0/3640396706 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f707006a670 0x7f707014ae40 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:39.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.111+0000 7f70774ff700 1 --2- 172.21.15.88:0/3640396706 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f70700c0b70 0x7f707014ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:39.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.111+0000 7f70774ff700 1 -- 172.21.15.88:0/3640396706 >> 172.21.15.88:0/3640396706 conn(0x7f7070065cd0 msgr2=0x7f7070149c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:39.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.111+0000 7f70774ff700 1 -- 172.21.15.88:0/3640396706 shutdown_connections 2023-10-18T16:07:39.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.112+0000 7f70774ff700 1 -- 172.21.15.88:0/3640396706 wait complete. 2023-10-18T16:07:39.127 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-18T16:07:39.127 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 64 --pg_num_min 64 2023-10-18T16:07:39.244 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.241+0000 7f8b71f0f700 1 Processor -- start 2023-10-18T16:07:39.245 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.241+0000 7f8b71f0f700 1 -- start start 2023-10-18T16:07:39.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.242+0000 7f8b71f0f700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8b6c14ba80 0x7f8b6c0bfce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:39.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.242+0000 7f8b71f0f700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8b6c0bf420 0x7f8b6c152130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:39.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.242+0000 7f8b71f0f700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b6c14f4b0 con 0x7f8b6c152620 2023-10-18T16:07:39.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.242+0000 7f8b71f0f700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b6c0bd9b0 con 0x7f8b6c0bf420 2023-10-18T16:07:39.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.242+0000 7f8b71f0f700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b6c14e530 con 0x7f8b6c14ba80 2023-10-18T16:07:39.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.242+0000 7f8b6affd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8b6c0bf420 0x7f8b6c152130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:39.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.242+0000 7f8b6affd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8b6c0bf420 0x7f8b6c152130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43890/0 (socket says 172.21.15.88:43890) 2023-10-18T16:07:39.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.242+0000 7f8b6affd700 1 -- 172.21.15.88:0/3319443711 learned_addr learned my addr 172.21.15.88:0/3319443711 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:39.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.242+0000 7f8b6b7fe700 1 --2- 172.21.15.88:0/3319443711 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8b6c14ba80 0x7f8b6c0bfce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:39.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.242+0000 7f8b6affd700 1 -- 172.21.15.88:0/3319443711 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8b6c14ba80 msgr2=0x7f8b6c0bfce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:39.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.242+0000 7f8b6affd700 1 --2- 172.21.15.88:0/3319443711 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8b6c14ba80 0x7f8b6c0bfce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:39.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.242+0000 7f8b6affd700 1 -- 172.21.15.88:0/3319443711 >> v1:172.21.15.151:6789/0 conn(0x7f8b6c152620 legacy=0x7f8b6c0be1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:39.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.242+0000 7f8b6affd700 1 -- 172.21.15.88:0/3319443711 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b6c0bbec0 con 0x7f8b6c0bf420 2023-10-18T16:07:39.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.242+0000 7f8b6b7fe700 1 --2- 172.21.15.88:0/3319443711 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8b6c14ba80 0x7f8b6c0bfce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-10-18T16:07:39.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.242+0000 7f8b6affd700 1 --2- 172.21.15.88:0/3319443711 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b6c0bf420 0x7f8b6c152130 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f8b60006fd0 tx=0x7f8b600090f0).ready entity=mon.0 client_cookie=5d22da46c1c6d3a8 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:39.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.242+0000 7f8b6a7fc700 1 -- 172.21.15.88:0/3319443711 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b6000b600 con 0x7f8b6c0bf420 2023-10-18T16:07:39.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.243+0000 7f8b6a7fc700 1 -- 172.21.15.88:0/3319443711 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b6000b760 con 0x7f8b6c0bf420 2023-10-18T16:07:39.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.243+0000 7f8b6a7fc700 1 -- 172.21.15.88:0/3319443711 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b6000bae0 con 0x7f8b6c0bf420 2023-10-18T16:07:39.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.243+0000 7f8b71f0f700 1 -- 172.21.15.88:0/3319443711 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b6c0bf420 msgr2=0x7f8b6c152130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:39.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.243+0000 7f8b71f0f700 1 --2- 172.21.15.88:0/3319443711 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b6c0bf420 0x7f8b6c152130 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f8b60006fd0 tx=0x7f8b600090f0).stop 2023-10-18T16:07:39.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.243+0000 7f8b71f0f700 1 -- 172.21.15.88:0/3319443711 shutdown_connections 2023-10-18T16:07:39.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.243+0000 7f8b71f0f700 1 --2- 172.21.15.88:0/3319443711 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8b6c14ba80 0x7f8b6c0bfce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:39.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.243+0000 7f8b71f0f700 1 --2- 172.21.15.88:0/3319443711 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b6c0bf420 0x7f8b6c152130 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:39.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.243+0000 7f8b71f0f700 1 -- 172.21.15.88:0/3319443711 >> 172.21.15.88:0/3319443711 conn(0x7f8b6c062650 msgr2=0x7f8b6c062a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:39.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.243+0000 7f8b71f0f700 1 -- 172.21.15.88:0/3319443711 shutdown_connections 2023-10-18T16:07:39.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.243+0000 7f8b71f0f700 1 -- 172.21.15.88:0/3319443711 wait complete. 2023-10-18T16:07:39.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.244+0000 7f8b71f0f700 1 Processor -- start 2023-10-18T16:07:39.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.244+0000 7f8b71f0f700 1 -- start start 2023-10-18T16:07:39.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.245+0000 7f8b71f0f700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b6c0bf420 0x7f8b6c162450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:39.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.245+0000 7f8b71f0f700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8b6c14ba80 0x7f8b6c1510a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:39.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.245+0000 7f8b71f0f700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8b6c152620 0x7f8b6c14c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:39.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.245+0000 7f8b71f0f700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b6c15f240 con 0x7f8b6c0bf420 2023-10-18T16:07:39.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.245+0000 7f8b71f0f700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b6c162940 con 0x7f8b6c14ba80 2023-10-18T16:07:39.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.245+0000 7f8b71f0f700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b6c15e600 con 0x7f8b6c152620 2023-10-18T16:07:39.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.245+0000 7f8b6b7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b6c0bf420 0x7f8b6c162450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:39.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.245+0000 7f8b6b7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b6c0bf420 0x7f8b6c162450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43894/0 (socket says 172.21.15.88:43894) 2023-10-18T16:07:39.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.245+0000 7f8b6b7fe700 1 -- 172.21.15.88:0/2533938914 learned_addr learned my addr 172.21.15.88:0/2533938914 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:39.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.245+0000 7f8b6affd700 1 --2- 172.21.15.88:0/2533938914 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8b6c14ba80 0x7f8b6c1510a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:39.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.245+0000 7f8b6bfff700 1 --2- 172.21.15.88:0/2533938914 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8b6c152620 0x7f8b6c14c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:39.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.245+0000 7f8b6b7fe700 1 -- 172.21.15.88:0/2533938914 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8b6c152620 msgr2=0x7f8b6c14c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:39.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.245+0000 7f8b6b7fe700 1 --2- 172.21.15.88:0/2533938914 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8b6c152620 0x7f8b6c14c0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:39.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.245+0000 7f8b6b7fe700 1 -- 172.21.15.88:0/2533938914 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8b6c14ba80 msgr2=0x7f8b6c1510a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:39.258 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.245+0000 7f8b6b7fe700 1 --2- 172.21.15.88:0/2533938914 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8b6c14ba80 0x7f8b6c1510a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:39.258 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.245+0000 7f8b6b7fe700 1 -- 172.21.15.88:0/2533938914 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b60009370 con 0x7f8b6c0bf420 2023-10-18T16:07:39.258 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.245+0000 7f8b6affd700 1 --2- 172.21.15.88:0/2533938914 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8b6c14ba80 0x7f8b6c1510a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:07:39.258 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.245+0000 7f8b6b7fe700 1 --2- 172.21.15.88:0/2533938914 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b6c0bf420 0x7f8b6c162450 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f8b5c00c8b0 tx=0x7f8b5c008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:39.258 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.246+0000 7f8b537fe700 1 -- 172.21.15.88:0/2533938914 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b5c010580 con 0x7f8b6c0bf420 2023-10-18T16:07:39.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.246+0000 7f8b537fe700 1 -- 172.21.15.88:0/2533938914 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b5c0106e0 con 0x7f8b6c0bf420 2023-10-18T16:07:39.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.246+0000 7f8b71f0f700 1 -- 172.21.15.88:0/2533938914 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b6c151590 con 0x7f8b6c0bf420 2023-10-18T16:07:39.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.246+0000 7f8b537fe700 1 -- 172.21.15.88:0/2533938914 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b5c010a60 con 0x7f8b6c0bf420 2023-10-18T16:07:39.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.246+0000 7f8b71f0f700 1 -- 172.21.15.88:0/2533938914 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b6c15eb80 con 0x7f8b6c0bf420 2023-10-18T16:07:39.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.247+0000 7f8b71f0f700 1 -- 172.21.15.88:0/2533938914 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b6c15f380 con 0x7f8b6c0bf420 2023-10-18T16:07:39.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.247+0000 7f8b537fe700 1 -- 172.21.15.88:0/2533938914 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f8b5c01b460 con 0x7f8b6c0bf420 2023-10-18T16:07:39.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.247+0000 7f8b537fe700 1 --2- 172.21.15.88:0/2533938914 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f8b5805f150 0x7f8b58061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:39.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.247+0000 7f8b537fe700 1 -- 172.21.15.88:0/2533938914 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5269+0+0 (secure 0 0 0) 0x7f8b5c079ec0 con 0x7f8b6c0bf420 2023-10-18T16:07:39.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.247+0000 7f8b6affd700 1 --2- 172.21.15.88:0/2533938914 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f8b5805f150 0x7f8b58061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:39.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.248+0000 7f8b6affd700 1 --2- 172.21.15.88:0/2533938914 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f8b5805f150 0x7f8b58061610 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f8b60006fd0 tx=0x7f8b60017000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:39.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.251+0000 7f8b537fe700 1 -- 172.21.15.88:0/2533938914 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f8b5c04b480 con 0x7f8b6c0bf420 2023-10-18T16:07:39.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:39.417+0000 7f8b71f0f700 1 -- 172.21.15.88:0/2533938914 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 64, "pg_num_min": 64} v 0) v1 -- 0x7f8b6c066e60 con 0x7f8b6c0bf420 2023-10-18T16:07:40.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.409+0000 7f8b537fe700 1 -- 172.21.15.88:0/2533938914 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 64, "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v19) v1 ==== 152+0+0 (secure 0 0 0) 0x7f8b5c04eaa0 con 0x7f8b6c0bf420 2023-10-18T16:07:40.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.412+0000 7f8b71f0f700 1 -- 172.21.15.88:0/2533938914 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f8b5805f150 msgr2=0x7f8b58061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:40.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.412+0000 7f8b71f0f700 1 --2- 172.21.15.88:0/2533938914 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f8b5805f150 0x7f8b58061610 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f8b60006fd0 tx=0x7f8b60017000).stop 2023-10-18T16:07:40.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.412+0000 7f8b71f0f700 1 -- 172.21.15.88:0/2533938914 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b6c0bf420 msgr2=0x7f8b6c162450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:40.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.412+0000 7f8b71f0f700 1 --2- 172.21.15.88:0/2533938914 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b6c0bf420 0x7f8b6c162450 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f8b5c00c8b0 tx=0x7f8b5c008040).stop 2023-10-18T16:07:40.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.412+0000 7f8b71f0f700 1 -- 172.21.15.88:0/2533938914 shutdown_connections 2023-10-18T16:07:40.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.412+0000 7f8b71f0f700 1 --2- 172.21.15.88:0/2533938914 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f8b5805f150 0x7f8b58061610 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:40.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.412+0000 7f8b71f0f700 1 --2- 172.21.15.88:0/2533938914 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8b6c152620 0x7f8b6c14c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:40.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.412+0000 7f8b71f0f700 1 --2- 172.21.15.88:0/2533938914 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b6c0bf420 0x7f8b6c162450 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:40.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.412+0000 7f8b71f0f700 1 --2- 172.21.15.88:0/2533938914 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8b6c14ba80 0x7f8b6c1510a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:40.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.412+0000 7f8b71f0f700 1 -- 172.21.15.88:0/2533938914 >> 172.21.15.88:0/2533938914 conn(0x7f8b6c062650 msgr2=0x7f8b6c14af60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:40.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.412+0000 7f8b71f0f700 1 -- 172.21.15.88:0/2533938914 shutdown_connections 2023-10-18T16:07:40.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.413+0000 7f8b71f0f700 1 -- 172.21.15.88:0/2533938914 wait complete. 2023-10-18T16:07:40.423 INFO:teuthology.orchestra.run.smithi088.stderr:pool 'cephfs_metadata' created 2023-10-18T16:07:40.435 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-18T16:07:40.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.556+0000 7f9394b95700 1 Processor -- start 2023-10-18T16:07:40.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.556+0000 7f9394b95700 1 -- start start 2023-10-18T16:07:40.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.557+0000 7f9394b95700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9390068710 0x7f9390068af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:40.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.557+0000 7f9394b95700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f93900678d0 0x7f9390150150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:40.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.557+0000 7f9394b95700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9390066920 con 0x7f9390064130 2023-10-18T16:07:40.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.557+0000 7f9394b95700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93900674c0 con 0x7f93900678d0 2023-10-18T16:07:40.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.557+0000 7f9394b95700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9390150720 con 0x7f9390068710 2023-10-18T16:07:40.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.557+0000 7f938ffff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f93900678d0 0x7f9390150150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:40.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.557+0000 7f938effd700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f9390064130 0x7f9390066db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:52358/0 (socket says 172.21.15.88:52358) 2023-10-18T16:07:40.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.557+0000 7f938ffff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f93900678d0 0x7f9390150150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43910/0 (socket says 172.21.15.88:43910) 2023-10-18T16:07:40.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.557+0000 7f938effd700 1 -- 172.21.15.88:0/2734779402 learned_addr learned my addr 172.21.15.88:0/2734779402 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:07:40.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.558+0000 7f938f7fe700 1 --2- 172.21.15.88:0/2734779402 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9390068710 0x7f9390068af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:40.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.558+0000 7f938ffff700 1 -- 172.21.15.88:0/2734779402 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9390068710 msgr2=0x7f9390068af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:40.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.558+0000 7f938ffff700 1 --2- 172.21.15.88:0/2734779402 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9390068710 0x7f9390068af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:40.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.558+0000 7f938ffff700 1 -- 172.21.15.88:0/2734779402 >> v1:172.21.15.88:6789/0 conn(0x7f9390064130 legacy=0x7f9390066db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:40.566 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.558+0000 7f938e7fc700 1 -- 172.21.15.88:0/2734779402 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1262539226 0 0) 0x7f9390066920 con 0x7f9390064130 2023-10-18T16:07:40.566 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.558+0000 7f938ffff700 1 -- 172.21.15.88:0/2734779402 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9390150a00 con 0x7f93900678d0 2023-10-18T16:07:40.566 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.558+0000 7f938f7fe700 1 --2- 172.21.15.88:0/2734779402 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9390068710 0x7f9390068af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:07:40.566 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.558+0000 7f938ffff700 1 --2- 172.21.15.88:0/2734779402 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f93900678d0 0x7f9390150150 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f9388009760 tx=0x7f9388009060).ready entity=mon.0 client_cookie=d6bd215adac0a210 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:40.567 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.558+0000 7f938e7fc700 1 -- 172.21.15.88:0/2734779402 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f938800b610 con 0x7f93900678d0 2023-10-18T16:07:40.567 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.558+0000 7f938e7fc700 1 -- 172.21.15.88:0/2734779402 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f938800b770 con 0x7f93900678d0 2023-10-18T16:07:40.567 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.558+0000 7f938e7fc700 1 -- 172.21.15.88:0/2734779402 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f938800baf0 con 0x7f93900678d0 2023-10-18T16:07:40.567 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.558+0000 7f9394b95700 1 -- 172.21.15.88:0/2734779402 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f93900678d0 msgr2=0x7f9390150150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:40.567 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.558+0000 7f9394b95700 1 --2- 172.21.15.88:0/2734779402 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f93900678d0 0x7f9390150150 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f9388009760 tx=0x7f9388009060).stop 2023-10-18T16:07:40.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.558+0000 7f9394b95700 1 -- 172.21.15.88:0/2734779402 shutdown_connections 2023-10-18T16:07:40.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.558+0000 7f9394b95700 1 --2- 172.21.15.88:0/2734779402 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f93900678d0 0x7f9390150150 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:40.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.558+0000 7f9394b95700 1 --2- 172.21.15.88:0/2734779402 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9390068710 0x7f9390068af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:40.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.559+0000 7f9394b95700 1 -- 172.21.15.88:0/2734779402 >> 172.21.15.88:0/2734779402 conn(0x7f93900c4770 msgr2=0x7f93900c4b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:40.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.559+0000 7f9394b95700 1 -- 172.21.15.88:0/2734779402 shutdown_connections 2023-10-18T16:07:40.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.559+0000 7f9394b95700 1 -- 172.21.15.88:0/2734779402 wait complete. 2023-10-18T16:07:40.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.560+0000 7f9394b95700 1 Processor -- start 2023-10-18T16:07:40.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.560+0000 7f9394b95700 1 -- start start 2023-10-18T16:07:40.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.560+0000 7f9394b95700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9390064130 0x7f93900c4d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:40.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.560+0000 7f9394b95700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f93900678d0 0x7f93901635e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:40.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.560+0000 7f9394b95700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9390068710 0x7f9390163ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:40.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.560+0000 7f9394b95700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93901521d0 con 0x7f9390068710 2023-10-18T16:07:40.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.560+0000 7f9394b95700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93900bc490 con 0x7f9390064130 2023-10-18T16:07:40.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.560+0000 7f9394b95700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93900bc730 con 0x7f93900678d0 2023-10-18T16:07:40.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.561+0000 7f938ffff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9390068710 0x7f9390163ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:40.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.561+0000 7f938ffff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9390068710 0x7f9390163ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43926/0 (socket says 172.21.15.88:43926) 2023-10-18T16:07:40.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.561+0000 7f938ffff700 1 -- 172.21.15.88:0/1962229533 learned_addr learned my addr 172.21.15.88:0/1962229533 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:40.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.561+0000 7f938f7fe700 1 --2- 172.21.15.88:0/1962229533 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9390064130 0x7f93900c4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:40.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.561+0000 7f938effd700 1 --2- 172.21.15.88:0/1962229533 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f93900678d0 0x7f93901635e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:40.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.561+0000 7f938ffff700 1 -- 172.21.15.88:0/1962229533 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f93900678d0 msgr2=0x7f93901635e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:40.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.561+0000 7f938ffff700 1 --2- 172.21.15.88:0/1962229533 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f93900678d0 0x7f93901635e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:40.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.561+0000 7f938ffff700 1 -- 172.21.15.88:0/1962229533 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9390064130 msgr2=0x7f93900c4d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:40.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.561+0000 7f938ffff700 1 --2- 172.21.15.88:0/1962229533 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9390064130 0x7f93900c4d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:40.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.561+0000 7f938ffff700 1 -- 172.21.15.88:0/1962229533 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93880092a0 con 0x7f9390068710 2023-10-18T16:07:40.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.561+0000 7f938ffff700 1 --2- 172.21.15.88:0/1962229533 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9390068710 0x7f9390163ad0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f939006a450 tx=0x7f9388017000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:40.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.561+0000 7f93777fe700 1 -- 172.21.15.88:0/1962229533 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f938800baf0 con 0x7f9390068710 2023-10-18T16:07:40.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.561+0000 7f93777fe700 1 -- 172.21.15.88:0/1962229533 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f938801b410 con 0x7f9390068710 2023-10-18T16:07:40.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.561+0000 7f9394b95700 1 -- 172.21.15.88:0/1962229533 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93900bd270 con 0x7f9390068710 2023-10-18T16:07:40.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.561+0000 7f93777fe700 1 -- 172.21.15.88:0/1962229533 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f938801b7d0 con 0x7f9390068710 2023-10-18T16:07:40.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.561+0000 7f9394b95700 1 -- 172.21.15.88:0/1962229533 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93900bd830 con 0x7f9390068710 2023-10-18T16:07:40.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.562+0000 7f9394b95700 1 -- 172.21.15.88:0/1962229533 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9390069b50 con 0x7f9390068710 2023-10-18T16:07:40.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.562+0000 7f93777fe700 1 -- 172.21.15.88:0/1962229533 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f938801bc00 con 0x7f9390068710 2023-10-18T16:07:40.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.563+0000 7f93777fe700 1 --2- 172.21.15.88:0/1962229533 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f937c05f180 0x7f937c061640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:40.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.563+0000 7f93777fe700 1 -- 172.21.15.88:0/1962229533 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5639+0+0 (secure 0 0 0) 0x7f938807bae0 con 0x7f9390068710 2023-10-18T16:07:40.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.563+0000 7f938f7fe700 1 --2- 172.21.15.88:0/1962229533 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f937c05f180 0x7f937c061640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:40.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.563+0000 7f938f7fe700 1 --2- 172.21.15.88:0/1962229533 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f937c05f180 0x7f937c061640 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f9380007600 tx=0x7f9380009850).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:40.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.567+0000 7f93777fe700 1 -- 172.21.15.88:0/1962229533 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f938804cf30 con 0x7f9390068710 2023-10-18T16:07:40.734 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:40.731+0000 7f9394b95700 1 -- 172.21.15.88:0/1962229533 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f9390150ce0 con 0x7f9390068710 2023-10-18T16:07:41.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.414+0000 7f93777fe700 1 -- 172.21.15.88:0/1962229533 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v20) v1 ==== 170+0+0 (secure 0 0 0) 0x7f9388050550 con 0x7f9390068710 2023-10-18T16:07:41.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.418+0000 7f9394b95700 1 -- 172.21.15.88:0/1962229533 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f937c05f180 msgr2=0x7f937c061640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:41.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.418+0000 7f9394b95700 1 --2- 172.21.15.88:0/1962229533 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f937c05f180 0x7f937c061640 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f9380007600 tx=0x7f9380009850).stop 2023-10-18T16:07:41.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.418+0000 7f9394b95700 1 -- 172.21.15.88:0/1962229533 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9390068710 msgr2=0x7f9390163ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:41.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.418+0000 7f9394b95700 1 --2- 172.21.15.88:0/1962229533 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9390068710 0x7f9390163ad0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f939006a450 tx=0x7f9388017000).stop 2023-10-18T16:07:41.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.418+0000 7f9394b95700 1 -- 172.21.15.88:0/1962229533 shutdown_connections 2023-10-18T16:07:41.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.418+0000 7f9394b95700 1 --2- 172.21.15.88:0/1962229533 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f937c05f180 0x7f937c061640 secure :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0x7f9380007600 tx=0x7f9380009850).stop 2023-10-18T16:07:41.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.418+0000 7f9394b95700 1 --2- 172.21.15.88:0/1962229533 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9390068710 0x7f9390163ad0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:41.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.418+0000 7f9394b95700 1 --2- 172.21.15.88:0/1962229533 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9390064130 0x7f93900c4d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:41.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.418+0000 7f9394b95700 1 --2- 172.21.15.88:0/1962229533 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f93900678d0 0x7f93901635e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:41.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.418+0000 7f9394b95700 1 -- 172.21.15.88:0/1962229533 >> 172.21.15.88:0/1962229533 conn(0x7f93900c4770 msgr2=0x7f939014b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:41.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.419+0000 7f9394b95700 1 -- 172.21.15.88:0/1962229533 shutdown_connections 2023-10-18T16:07:41.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.419+0000 7f9394b95700 1 -- 172.21.15.88:0/1962229533 wait complete. 2023-10-18T16:07:41.438 INFO:teuthology.orchestra.run.smithi088.stderr:pool 'cephfs_data' created 2023-10-18T16:07:41.451 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-18T16:07:41.567 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.564+0000 7fba69df9700 1 Processor -- start 2023-10-18T16:07:41.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.564+0000 7fba69df9700 1 -- start start 2023-10-18T16:07:41.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.565+0000 7fba69df9700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fba64063d70 0x7fba640c6e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:41.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.565+0000 7fba69df9700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fba640c2400 0x7fba640c2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:41.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.565+0000 7fba69df9700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba64151020 con 0x7fba640c6a60 2023-10-18T16:07:41.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.565+0000 7fba69df9700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba64150090 con 0x7fba640c2400 2023-10-18T16:07:41.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.565+0000 7fba69df9700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba640c2d80 con 0x7fba64063d70 2023-10-18T16:07:41.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.565+0000 7fba695f8700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fba640c2400 0x7fba640c2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:41.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.565+0000 7fba63fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fba64063d70 0x7fba640c6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:41.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.565+0000 7fba695f8700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fba640c2400 0x7fba640c2890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.88:57550/0 (socket says 172.21.15.88:57550) 2023-10-18T16:07:41.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.565+0000 7fba695f8700 1 -- 172.21.15.88:0/2830393542 learned_addr learned my addr 172.21.15.88:0/2830393542 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:41.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.565+0000 7fba68df7700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fba640c6a60 0x7fba64153660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.88:59976/0 (socket says 172.21.15.88:59976) 2023-10-18T16:07:41.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.565+0000 7fba63fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fba64063d70 0x7fba640c6e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.88:51172/0 (socket says 172.21.15.88:51172) 2023-10-18T16:07:41.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.565+0000 7fba695f8700 1 -- 172.21.15.88:0/2830393542 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fba64063d70 msgr2=0x7fba640c6e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:41.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.565+0000 7fba695f8700 1 --2- 172.21.15.88:0/2830393542 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fba64063d70 0x7fba640c6e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:41.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.565+0000 7fba695f8700 1 -- 172.21.15.88:0/2830393542 >> v1:172.21.15.151:6789/0 conn(0x7fba640c6a60 legacy=0x7fba64153660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:41.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.565+0000 7fba637fe700 1 -- 172.21.15.88:0/2830393542 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2132463336 0 0) 0x7fba58001110 con 0x7fba640c6a60 2023-10-18T16:07:41.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.565+0000 7fba695f8700 1 -- 172.21.15.88:0/2830393542 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba64150b90 con 0x7fba640c2400 2023-10-18T16:07:41.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.566+0000 7fba695f8700 1 --2- 172.21.15.88:0/2830393542 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fba640c2400 0x7fba640c2890 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fba5c009f50 tx=0x7fba5c009840).ready entity=mon.1 client_cookie=8a9e38d608d44997 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:41.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.566+0000 7fba637fe700 1 -- 172.21.15.88:0/2830393542 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba5c00e070 con 0x7fba640c2400 2023-10-18T16:07:41.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.566+0000 7fba637fe700 1 -- 172.21.15.88:0/2830393542 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba5c00b9c0 con 0x7fba640c2400 2023-10-18T16:07:41.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.566+0000 7fba637fe700 1 -- 172.21.15.88:0/2830393542 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba5c00bd40 con 0x7fba640c2400 2023-10-18T16:07:41.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.566+0000 7fba69df9700 1 -- 172.21.15.88:0/2830393542 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fba640c2400 msgr2=0x7fba640c2890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:41.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.566+0000 7fba69df9700 1 --2- 172.21.15.88:0/2830393542 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fba640c2400 0x7fba640c2890 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fba5c009f50 tx=0x7fba5c009840).stop 2023-10-18T16:07:41.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.566+0000 7fba69df9700 1 -- 172.21.15.88:0/2830393542 shutdown_connections 2023-10-18T16:07:41.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.566+0000 7fba69df9700 1 --2- 172.21.15.88:0/2830393542 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fba640c2400 0x7fba640c2890 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:41.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.566+0000 7fba69df9700 1 --2- 172.21.15.88:0/2830393542 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fba64063d70 0x7fba640c6e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:41.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.566+0000 7fba69df9700 1 -- 172.21.15.88:0/2830393542 >> 172.21.15.88:0/2830393542 conn(0x7fba6405e050 msgr2=0x7fba6405e460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:41.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.566+0000 7fba69df9700 1 -- 172.21.15.88:0/2830393542 shutdown_connections 2023-10-18T16:07:41.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.566+0000 7fba69df9700 1 -- 172.21.15.88:0/2830393542 wait complete. 2023-10-18T16:07:41.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.567+0000 7fba69df9700 1 Processor -- start 2023-10-18T16:07:41.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.568+0000 7fba69df9700 1 -- start start 2023-10-18T16:07:41.579 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.568+0000 7fba69df9700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fba64063d70 0x7fba6414b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:41.579 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.568+0000 7fba69df9700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fba640c2400 0x7fba64152530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:41.579 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.568+0000 7fba69df9700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba640c6a60 0x7fba640c58a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:41.579 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.568+0000 7fba69df9700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba641542d0 con 0x7fba640c6a60 2023-10-18T16:07:41.579 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.568+0000 7fba69df9700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba64160250 con 0x7fba64063d70 2023-10-18T16:07:41.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.568+0000 7fba69df9700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba6414b550 con 0x7fba640c2400 2023-10-18T16:07:41.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.568+0000 7fba695f8700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba640c6a60 0x7fba640c58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:41.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.568+0000 7fba695f8700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba640c6a60 0x7fba640c58a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43932/0 (socket says 172.21.15.88:43932) 2023-10-18T16:07:41.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.568+0000 7fba695f8700 1 -- 172.21.15.88:0/4217617366 learned_addr learned my addr 172.21.15.88:0/4217617366 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:41.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.569+0000 7fba68df7700 1 --2- 172.21.15.88:0/4217617366 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fba64063d70 0x7fba6414b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:41.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.569+0000 7fba63fff700 1 --2- 172.21.15.88:0/4217617366 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fba640c2400 0x7fba64152530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:41.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.569+0000 7fba695f8700 1 -- 172.21.15.88:0/4217617366 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fba640c2400 msgr2=0x7fba64152530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:41.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.569+0000 7fba695f8700 1 --2- 172.21.15.88:0/4217617366 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fba640c2400 0x7fba64152530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:41.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.569+0000 7fba695f8700 1 -- 172.21.15.88:0/4217617366 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fba64063d70 msgr2=0x7fba6414b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:41.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.569+0000 7fba695f8700 1 --2- 172.21.15.88:0/4217617366 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fba64063d70 0x7fba6414b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:41.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.569+0000 7fba695f8700 1 -- 172.21.15.88:0/4217617366 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba5c009ae0 con 0x7fba640c6a60 2023-10-18T16:07:41.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.569+0000 7fba695f8700 1 --2- 172.21.15.88:0/4217617366 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba640c6a60 0x7fba640c58a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fba5c00b6c0 tx=0x7fba640c3210).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:41.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.569+0000 7fba60ff9700 1 -- 172.21.15.88:0/4217617366 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba5c0092e0 con 0x7fba640c6a60 2023-10-18T16:07:41.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.569+0000 7fba60ff9700 1 -- 172.21.15.88:0/4217617366 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba5c01e470 con 0x7fba640c6a60 2023-10-18T16:07:41.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.569+0000 7fba69df9700 1 -- 172.21.15.88:0/4217617366 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba6414c110 con 0x7fba640c6a60 2023-10-18T16:07:41.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.569+0000 7fba60ff9700 1 -- 172.21.15.88:0/4217617366 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba5c01e830 con 0x7fba640c6a60 2023-10-18T16:07:41.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.569+0000 7fba69df9700 1 -- 172.21.15.88:0/4217617366 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba64153ff0 con 0x7fba640c6a60 2023-10-18T16:07:41.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.570+0000 7fba69df9700 1 -- 172.21.15.88:0/4217617366 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba64084340 con 0x7fba640c6a60 2023-10-18T16:07:41.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.570+0000 7fba60ff9700 1 -- 172.21.15.88:0/4217617366 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7fba5c003040 con 0x7fba640c6a60 2023-10-18T16:07:41.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.571+0000 7fba60ff9700 1 --2- 172.21.15.88:0/4217617366 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fba4805f150 0x7fba48061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:41.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.571+0000 7fba60ff9700 1 -- 172.21.15.88:0/4217617366 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6021+0+0 (secure 0 0 0) 0x7fba5c07b7b0 con 0x7fba640c6a60 2023-10-18T16:07:41.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.571+0000 7fba68df7700 1 --2- 172.21.15.88:0/4217617366 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fba4805f150 0x7fba48061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:41.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.571+0000 7fba68df7700 1 --2- 172.21.15.88:0/4217617366 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fba4805f150 0x7fba48061610 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7fba58007750 tx=0x7fba58007c90).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:41.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.575+0000 7fba60ff9700 1 -- 172.21.15.88:0/4217617366 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7fba5c04cb80 con 0x7fba640c6a60 2023-10-18T16:07:41.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.780+0000 7fba69df9700 1 -- 172.21.15.88:0/4217617366 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fba64160390 con 0x7fba640c6a60 2023-10-18T16:07:41.786 INFO:tasks.ceph.mds.d.smithi151.stderr:terminate called after throwing an instance of 'boost::wrapexcept' 2023-10-18T16:07:41.787 INFO:tasks.ceph.mds.d.smithi151.stderr: what(): boost::bad_get: failed value get using boost::get 2023-10-18T16:07:41.787 INFO:tasks.ceph.mds.d.smithi151.stderr:*** Caught signal (Aborted) ** 2023-10-18T16:07:41.787 INFO:tasks.ceph.mds.d.smithi151.stderr: in thread 7faf43e18700 thread_name:ms_dispatch 2023-10-18T16:07:41.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.785+0000 7fba60ff9700 1 -- 172.21.15.88:0/4217617366 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 3 and data pool 4 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7fba5c023070 con 0x7fba640c6a60 2023-10-18T16:07:41.789 INFO:tasks.ceph.mds.d.smithi151.stderr: ceph version 16.2.14-100-gb4ddd52c (b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5) pacific (stable) 2023-10-18T16:07:41.789 INFO:tasks.ceph.mds.d.smithi151.stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7faf4b61bb20] 2023-10-18T16:07:41.789 INFO:tasks.ceph.mds.d.smithi151.stderr: 2: gsignal() 2023-10-18T16:07:41.790 INFO:tasks.ceph.mds.d.smithi151.stderr: 3: abort() 2023-10-18T16:07:41.790 INFO:tasks.ceph.mds.d.smithi151.stderr: 4: /lib64/libstdc++.so.6(+0x9009b) [0x7faf4aa1c09b] 2023-10-18T16:07:41.790 INFO:tasks.ceph.mds.d.smithi151.stderr: 5: /lib64/libstdc++.so.6(+0x9653c) [0x7faf4aa2253c] 2023-10-18T16:07:41.790 INFO:tasks.ceph.mds.d.smithi151.stderr: 6: /lib64/libstdc++.so.6(+0x96597) [0x7faf4aa22597] 2023-10-18T16:07:41.790 INFO:tasks.ceph.mds.d.smithi151.stderr: 7: /lib64/libstdc++.so.6(+0x967f8) [0x7faf4aa227f8] 2023-10-18T16:07:41.791 INFO:tasks.ceph.mds.d.smithi151.stderr: 8: ceph-mds(+0x212b43) [0x55a847977b43] 2023-10-18T16:07:41.791 INFO:tasks.ceph.mds.d.smithi151.stderr: 9: (Option::size_t const md_config_t::get_val(ConfigValues const&, std::basic_string_view >) const+0x5c) [0x55a8479ec89c] 2023-10-18T16:07:41.791 INFO:tasks.ceph.mds.d.smithi151.stderr: 10: (SessionMap::SessionMap(MDSRank*)+0x2c2) [0x55a847c346a2] 2023-10-18T16:07:41.791 INFO:tasks.ceph.mds.d.smithi151.stderr: 11: (MDSRank::MDSRank(int, std::__cxx11::basic_string, std::allocator >, ceph::fair_mutex&, std::shared_ptr&, CommonSafeTimer&, Beacon&, std::unique_ptr >&, Messenger*, MonClient*, MgrClient*, Context*, Context*, boost::asio::io_context&)+0x214) [0x55a847951724] 2023-10-18T16:07:41.791 INFO:tasks.ceph.mds.d.smithi151.stderr: 12: (MDSRankDispatcher::MDSRankDispatcher(int, std::__cxx11::basic_string, std::allocator >, ceph::fair_mutex&, std::shared_ptr&, CommonSafeTimer&, Beacon&, std::unique_ptr >&, Messenger*, MonClient*, MgrClient*, Context*, Context*, boost::asio::io_context&)+0x101) [0x55a847953181] 2023-10-18T16:07:41.792 INFO:tasks.ceph.mds.d.smithi151.stderr: 13: (MDSDaemon::handle_mds_map(boost::intrusive_ptr const&)+0x1608) [0x55a847923348] 2023-10-18T16:07:41.792 INFO:tasks.ceph.mds.d.smithi151.stderr: 14: (MDSDaemon::handle_core_message(boost::intrusive_ptr const&)+0xcd) [0x55a84792644d] 2023-10-18T16:07:41.792 INFO:tasks.ceph.mds.d.smithi151.stderr: 15: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0xc3) [0x55a847926fa3] 2023-10-18T16:07:41.792 INFO:tasks.ceph.mds.d.smithi151.stderr: 16: (DispatchQueue::entry()+0x126a) [0x7faf4c8754ea] 2023-10-18T16:07:41.792 INFO:tasks.ceph.mds.d.smithi151.stderr: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7faf4c929d51] 2023-10-18T16:07:41.792 INFO:tasks.ceph.mds.d.smithi151.stderr: 18: /lib64/libpthread.so.0(+0x814a) [0x7faf4b61114a] 2023-10-18T16:07:41.793 INFO:tasks.ceph.mds.d.smithi151.stderr: 19: clone() 2023-10-18T16:07:41.793 INFO:tasks.ceph.mds.d.smithi151.stderr:2023-10-18T16:07:41.785+0000 7faf43e18700 -1 *** Caught signal (Aborted) ** 2023-10-18T16:07:41.793 INFO:tasks.ceph.mds.d.smithi151.stderr: in thread 7faf43e18700 thread_name:ms_dispatch 2023-10-18T16:07:41.793 INFO:tasks.ceph.mds.d.smithi151.stderr: 2023-10-18T16:07:41.793 INFO:tasks.ceph.mds.d.smithi151.stderr: ceph version 16.2.14-100-gb4ddd52c (b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5) pacific (stable) 2023-10-18T16:07:41.794 INFO:tasks.ceph.mds.d.smithi151.stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7faf4b61bb20] 2023-10-18T16:07:41.794 INFO:tasks.ceph.mds.d.smithi151.stderr: 2: gsignal() 2023-10-18T16:07:41.794 INFO:tasks.ceph.mds.d.smithi151.stderr: 3: abort() 2023-10-18T16:07:41.794 INFO:tasks.ceph.mds.d.smithi151.stderr: 4: /lib64/libstdc++.so.6(+0x9009b) [0x7faf4aa1c09b] 2023-10-18T16:07:41.794 INFO:tasks.ceph.mds.d.smithi151.stderr: 5: /lib64/libstdc++.so.6(+0x9653c) [0x7faf4aa2253c] 2023-10-18T16:07:41.795 INFO:tasks.ceph.mds.d.smithi151.stderr: 6: /lib64/libstdc++.so.6(+0x96597) [0x7faf4aa22597] 2023-10-18T16:07:41.795 INFO:tasks.ceph.mds.d.smithi151.stderr: 7: /lib64/libstdc++.so.6(+0x967f8) [0x7faf4aa227f8] 2023-10-18T16:07:41.795 INFO:tasks.ceph.mds.d.smithi151.stderr: 8: ceph-mds(+0x212b43) [0x55a847977b43] 2023-10-18T16:07:41.795 INFO:tasks.ceph.mds.d.smithi151.stderr: 9: (Option::size_t const md_config_t::get_val(ConfigValues const&, std::basic_string_view >) const+0x5c) [0x55a8479ec89c] 2023-10-18T16:07:41.795 INFO:tasks.ceph.mds.d.smithi151.stderr: 10: (SessionMap::SessionMap(MDSRank*)+0x2c2) [0x55a847c346a2] 2023-10-18T16:07:41.796 INFO:tasks.ceph.mds.d.smithi151.stderr: 11: (MDSRank::MDSRank(int, std::__cxx11::basic_string, std::allocator >, ceph::fair_mutex&, std::shared_ptr&, CommonSafeTimer&, Beacon&, std::unique_ptr >&, Messenger*, MonClient*, MgrClient*, Context*, Context*, boost::asio::io_context&)+0x214) [0x55a847951724] 2023-10-18T16:07:41.796 INFO:tasks.ceph.mds.d.smithi151.stderr: 12: (MDSRankDispatcher::MDSRankDispatcher(int, std::__cxx11::basic_string, std::allocator >, ceph::fair_mutex&, std::shared_ptr&, CommonSafeTimer&, Beacon&, std::unique_ptr >&, Messenger*, MonClient*, MgrClient*, Context*, Context*, boost::asio::io_context&)+0x101) [0x55a847953181] 2023-10-18T16:07:41.796 INFO:tasks.ceph.mds.d.smithi151.stderr: 13: (MDSDaemon::handle_mds_map(boost::intrusive_ptr const&)+0x1608) [0x55a847923348] 2023-10-18T16:07:41.796 INFO:tasks.ceph.mds.d.smithi151.stderr: 14: (MDSDaemon::handle_core_message(boost::intrusive_ptr const&)+0xcd) [0x55a84792644d] 2023-10-18T16:07:41.796 INFO:tasks.ceph.mds.d.smithi151.stderr: 15: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0xc3) [0x55a847926fa3] 2023-10-18T16:07:41.796 INFO:tasks.ceph.mds.d.smithi151.stderr: 16: (DispatchQueue::entry()+0x126a) [0x7faf4c8754ea] 2023-10-18T16:07:41.797 INFO:tasks.ceph.mds.d.smithi151.stderr: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7faf4c929d51] 2023-10-18T16:07:41.797 INFO:tasks.ceph.mds.d.smithi151.stderr: 18: /lib64/libpthread.so.0(+0x814a) [0x7faf4b61114a] 2023-10-18T16:07:41.797 INFO:tasks.ceph.mds.d.smithi151.stderr: 19: clone() 2023-10-18T16:07:41.797 INFO:tasks.ceph.mds.d.smithi151.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-10-18T16:07:41.797 INFO:tasks.ceph.mds.d.smithi151.stderr: 2023-10-18T16:07:41.798 INFO:tasks.ceph.mds.d.smithi151.stderr: 0> 2023-10-18T16:07:41.785+0000 7faf43e18700 -1 *** Caught signal (Aborted) ** 2023-10-18T16:07:41.799 INFO:tasks.ceph.mds.d.smithi151.stderr: in thread 7faf43e18700 thread_name:ms_dispatch 2023-10-18T16:07:41.799 INFO:tasks.ceph.mds.d.smithi151.stderr: 2023-10-18T16:07:41.799 INFO:tasks.ceph.mds.d.smithi151.stderr: ceph version 16.2.14-100-gb4ddd52c (b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5) pacific (stable) 2023-10-18T16:07:41.799 INFO:tasks.ceph.mds.d.smithi151.stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7faf4b61bb20] 2023-10-18T16:07:41.799 INFO:tasks.ceph.mds.d.smithi151.stderr: 2: gsignal() 2023-10-18T16:07:41.800 INFO:tasks.ceph.mds.d.smithi151.stderr: 3: abort() 2023-10-18T16:07:41.800 INFO:tasks.ceph.mds.d.smithi151.stderr: 4: /lib64/libstdc++.so.6(+0x9009b) [0x7faf4aa1c09b] 2023-10-18T16:07:41.800 INFO:tasks.ceph.mds.d.smithi151.stderr: 5: /lib64/libstdc++.so.6(+0x9653c) [0x7faf4aa2253c] 2023-10-18T16:07:41.800 INFO:tasks.ceph.mds.d.smithi151.stderr: 6: /lib64/libstdc++.so.6(+0x96597) [0x7faf4aa22597] 2023-10-18T16:07:41.800 INFO:tasks.ceph.mds.d.smithi151.stderr: 7: /lib64/libstdc++.so.6(+0x967f8) [0x7faf4aa227f8] 2023-10-18T16:07:41.801 INFO:tasks.ceph.mds.d.smithi151.stderr: 8: ceph-mds(+0x212b43) [0x55a847977b43] 2023-10-18T16:07:41.801 INFO:tasks.ceph.mds.d.smithi151.stderr: 9: (Option::size_t const md_config_t::get_val(ConfigValues const&, std::basic_string_view >) const+0x5c) [0x55a8479ec89c] 2023-10-18T16:07:41.801 INFO:tasks.ceph.mds.d.smithi151.stderr: 10: (SessionMap::SessionMap(MDSRank*)+0x2c2) [0x55a847c346a2] 2023-10-18T16:07:41.801 INFO:tasks.ceph.mds.d.smithi151.stderr: 11: (MDSRank::MDSRank(int, std::__cxx11::basic_string, std::allocator >, ceph::fair_mutex&, std::shared_ptr&, CommonSafeTimer&, Beacon&, std::unique_ptr >&, Messenger*, MonClient*, MgrClient*, Context*, Context*, boost::asio::io_context&)+0x214) [0x55a847951724] 2023-10-18T16:07:41.802 INFO:tasks.ceph.mds.d.smithi151.stderr: 12: (MDSRankDispatcher::MDSRankDispatcher(int, std::__cxx11::basic_string, std::allocator >, ceph::fair_mutex&, std::shared_ptr&, CommonSafeTimer&, Beacon&, std::unique_ptr >&, Messenger*, MonClient*, MgrClient*, Context*, Context*, boost::asio::io_context&)+0x101) [0x55a847953181] 2023-10-18T16:07:41.802 INFO:tasks.ceph.mds.d.smithi151.stderr: 13: (MDSDaemon::handle_mds_map(boost::intrusive_ptr const&)+0x1608) [0x55a847923348] 2023-10-18T16:07:41.802 INFO:tasks.ceph.mds.d.smithi151.stderr: 14: (MDSDaemon::handle_core_message(boost::intrusive_ptr const&)+0xcd) [0x55a84792644d] 2023-10-18T16:07:41.802 INFO:tasks.ceph.mds.d.smithi151.stderr: 15: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0xc3) [0x55a847926fa3] 2023-10-18T16:07:41.802 INFO:tasks.ceph.mds.d.smithi151.stderr: 16: (DispatchQueue::entry()+0x126a) [0x7faf4c8754ea] 2023-10-18T16:07:41.803 INFO:tasks.ceph.mds.d.smithi151.stderr: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7faf4c929d51] 2023-10-18T16:07:41.803 INFO:tasks.ceph.mds.d.smithi151.stderr: 18: /lib64/libpthread.so.0(+0x814a) [0x7faf4b61114a] 2023-10-18T16:07:41.803 INFO:tasks.ceph.mds.d.smithi151.stderr: 19: clone() 2023-10-18T16:07:41.803 INFO:tasks.ceph.mds.d.smithi151.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-10-18T16:07:41.803 INFO:tasks.ceph.mds.d.smithi151.stderr: 2023-10-18T16:07:41.804 INFO:tasks.ceph.mds.d.smithi151.stderr: 0> 2023-10-18T16:07:41.785+0000 7faf43e18700 -1 *** Caught signal (Aborted) ** 2023-10-18T16:07:41.804 INFO:tasks.ceph.mds.d.smithi151.stderr: in thread 7faf43e18700 thread_name:ms_dispatch 2023-10-18T16:07:41.804 INFO:tasks.ceph.mds.d.smithi151.stderr: 2023-10-18T16:07:41.804 INFO:tasks.ceph.mds.d.smithi151.stderr: ceph version 16.2.14-100-gb4ddd52c (b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5) pacific (stable) 2023-10-18T16:07:41.804 INFO:tasks.ceph.mds.d.smithi151.stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7faf4b61bb20] 2023-10-18T16:07:41.805 INFO:tasks.ceph.mds.d.smithi151.stderr: 2: gsignal() 2023-10-18T16:07:41.805 INFO:tasks.ceph.mds.d.smithi151.stderr: 3: abort() 2023-10-18T16:07:41.805 INFO:tasks.ceph.mds.d.smithi151.stderr: 4: /lib64/libstdc++.so.6(+0x9009b) [0x7faf4aa1c09b] 2023-10-18T16:07:41.805 INFO:tasks.ceph.mds.d.smithi151.stderr: 5: /lib64/libstdc++.so.6(+0x9653c) [0x7faf4aa2253c] 2023-10-18T16:07:41.805 INFO:tasks.ceph.mds.d.smithi151.stderr: 6: /lib64/libstdc++.so.6(+0x96597) [0x7faf4aa22597] 2023-10-18T16:07:41.806 INFO:tasks.ceph.mds.d.smithi151.stderr: 7: /lib64/libstdc++.so.6(+0x967f8) [0x7faf4aa227f8] 2023-10-18T16:07:41.806 INFO:tasks.ceph.mds.d.smithi151.stderr: 8: ceph-mds(+0x212b43) [0x55a847977b43] 2023-10-18T16:07:41.806 INFO:tasks.ceph.mds.d.smithi151.stderr: 9: (Option::size_t const md_config_t::get_val(ConfigValues const&, std::basic_string_view >) const+0x5c) [0x55a8479ec89c] 2023-10-18T16:07:41.806 INFO:tasks.ceph.mds.d.smithi151.stderr: 10: (SessionMap::SessionMap(MDSRank*)+0x2c2) [0x55a847c346a2] 2023-10-18T16:07:41.806 INFO:tasks.ceph.mds.d.smithi151.stderr: 11: (MDSRank::MDSRank(int, std::__cxx11::basic_string, std::allocator >, ceph::fair_mutex&, std::shared_ptr&, CommonSafeTimer&, Beacon&, std::unique_ptr >&, Messenger*, MonClient*, MgrClient*, Context*, Context*, boost::asio::io_context&)+0x214) [0x55a847951724] 2023-10-18T16:07:41.807 INFO:tasks.ceph.mds.d.smithi151.stderr: 12: (MDSRankDispatcher::MDSRankDispatcher(int, std::__cxx11::basic_string, std::allocator >, ceph::fair_mutex&, std::shared_ptr&, CommonSafeTimer&, Beacon&, std::unique_ptr >&, Messenger*, MonClient*, MgrClient*, Context*, Context*, boost::asio::io_context&)+0x101) [0x55a847953181] 2023-10-18T16:07:41.807 INFO:tasks.ceph.mds.d.smithi151.stderr: 13: (MDSDaemon::handle_mds_map(boost::intrusive_ptr const&)+0x1608) [0x55a847923348] 2023-10-18T16:07:41.807 INFO:tasks.ceph.mds.d.smithi151.stderr: 14: (MDSDaemon::handle_core_message(boost::intrusive_ptr const&)+0xcd) [0x55a84792644d] 2023-10-18T16:07:41.807 INFO:tasks.ceph.mds.d.smithi151.stderr: 15: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0xc3) [0x55a847926fa3] 2023-10-18T16:07:41.807 INFO:tasks.ceph.mds.d.smithi151.stderr: 16: (DispatchQueue::entry()+0x126a) [0x7faf4c8754ea] 2023-10-18T16:07:41.808 INFO:tasks.ceph.mds.d.smithi151.stderr: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7faf4c929d51] 2023-10-18T16:07:41.808 INFO:tasks.ceph.mds.d.smithi151.stderr: 18: /lib64/libpthread.so.0(+0x814a) [0x7faf4b61114a] 2023-10-18T16:07:41.808 INFO:tasks.ceph.mds.d.smithi151.stderr: 19: clone() 2023-10-18T16:07:41.808 INFO:tasks.ceph.mds.d.smithi151.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-10-18T16:07:41.808 INFO:tasks.ceph.mds.d.smithi151.stderr: 2023-10-18T16:07:41.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.790+0000 7fba69df9700 1 -- 172.21.15.88:0/4217617366 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fba4805f150 msgr2=0x7fba48061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:41.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.790+0000 7fba69df9700 1 --2- 172.21.15.88:0/4217617366 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fba4805f150 0x7fba48061610 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7fba58007750 tx=0x7fba58007c90).stop 2023-10-18T16:07:41.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.790+0000 7fba69df9700 1 -- 172.21.15.88:0/4217617366 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba640c6a60 msgr2=0x7fba640c58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:41.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.790+0000 7fba69df9700 1 --2- 172.21.15.88:0/4217617366 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba640c6a60 0x7fba640c58a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fba5c00b6c0 tx=0x7fba640c3210).stop 2023-10-18T16:07:41.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.790+0000 7fba69df9700 1 -- 172.21.15.88:0/4217617366 shutdown_connections 2023-10-18T16:07:41.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.790+0000 7fba69df9700 1 --2- 172.21.15.88:0/4217617366 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fba4805f150 0x7fba48061610 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:41.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.790+0000 7fba69df9700 1 --2- 172.21.15.88:0/4217617366 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba640c6a60 0x7fba640c58a0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:41.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.790+0000 7fba69df9700 1 --2- 172.21.15.88:0/4217617366 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fba64063d70 0x7fba6414b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:41.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.790+0000 7fba69df9700 1 --2- 172.21.15.88:0/4217617366 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fba640c2400 0x7fba64152530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:41.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.790+0000 7fba69df9700 1 -- 172.21.15.88:0/4217617366 >> 172.21.15.88:0/4217617366 conn(0x7fba6405e050 msgr2=0x7fba6414df30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:41.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.796+0000 7fba69df9700 1 -- 172.21.15.88:0/4217617366 shutdown_connections 2023-10-18T16:07:41.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.796+0000 7fba69df9700 1 -- 172.21.15.88:0/4217617366 wait complete. 2023-10-18T16:07:41.812 INFO:teuthology.orchestra.run.smithi088.stderr:new fs with metadata pool 3 and data pool 4 2023-10-18T16:07:41.818 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-18T16:07:41.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.927+0000 7fd0f6fd5700 1 Processor -- start 2023-10-18T16:07:41.931 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.927+0000 7fd0f6fd5700 1 -- start start 2023-10-18T16:07:41.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.928+0000 7fd0f6fd5700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fd0f00c2e70 0x7fd0f01503b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:41.935 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.928+0000 7fd0f6fd5700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd0f01508a0 0x7fd0f00c1f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:41.935 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.928+0000 7fd0f6fd5700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0f0150d30 con 0x7fd0f014d8f0 2023-10-18T16:07:41.935 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.928+0000 7fd0f6fd5700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0f00a7190 con 0x7fd0f00c2e70 2023-10-18T16:07:41.935 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.928+0000 7fd0f6fd5700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f00c3e10 con 0x7fd0f01508a0 2023-10-18T16:07:41.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.928+0000 7fd0f4d71700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fd0f014d8f0 0x7fd0f00a9080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:52366/0 (socket says 172.21.15.88:52366) 2023-10-18T16:07:41.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.928+0000 7fd0f4d71700 1 -- 172.21.15.88:0/1338414154 learned_addr learned my addr 172.21.15.88:0/1338414154 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:07:41.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.928+0000 7fd0e7fff700 1 --2- 172.21.15.88:0/1338414154 >> v2:172.21.15.151:3300/0 conn(0x7fd0f00c2e70 0x7fd0f01503b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:41.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.928+0000 7fd0f5572700 1 --2- 172.21.15.88:0/1338414154 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd0f01508a0 0x7fd0f00c1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:41.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.928+0000 7fd0e77fe700 1 -- 172.21.15.88:0/1338414154 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 921073670 0 0) 0x7fd0f0150d30 con 0x7fd0f014d8f0 2023-10-18T16:07:41.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.928+0000 7fd0e77fe700 1 -- 172.21.15.88:0/1338414154 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0d80033f0 con 0x7fd0f014d8f0 2023-10-18T16:07:41.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.928+0000 7fd0e77fe700 1 -- 172.21.15.88:0/1338414154 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4073665579 0 0) 0x7fd0d80033f0 con 0x7fd0f014d8f0 2023-10-18T16:07:41.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.928+0000 7fd0e77fe700 1 -- 172.21.15.88:0/1338414154 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd0f01508a0 msgr2=0x7fd0f00c1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:41.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.928+0000 7fd0e77fe700 1 --2- 172.21.15.88:0/1338414154 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd0f01508a0 0x7fd0f00c1f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:41.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.928+0000 7fd0e77fe700 1 -- 172.21.15.88:0/1338414154 >> v2:172.21.15.151:3300/0 conn(0x7fd0f00c2e70 msgr2=0x7fd0f01503b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:41.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.928+0000 7fd0e77fe700 1 --2- 172.21.15.88:0/1338414154 >> v2:172.21.15.151:3300/0 conn(0x7fd0f00c2e70 0x7fd0f01503b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:41.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.928+0000 7fd0e77fe700 1 -- 172.21.15.88:0/1338414154 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0f0151160 con 0x7fd0f014d8f0 2023-10-18T16:07:41.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.928+0000 7fd0e77fe700 1 -- 172.21.15.88:0/1338414154 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441715075 0 0) 0x7fd0ec001470 con 0x7fd0f014d8f0 2023-10-18T16:07:41.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.928+0000 7fd0e77fe700 1 -- 172.21.15.88:0/1338414154 >> v1:172.21.15.88:6789/0 conn(0x7fd0f014d8f0 legacy=0x7fd0f00a9080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:41.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.928+0000 7fd0e77fe700 1 --2- 172.21.15.88:0/1338414154 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd0f01508a0 0x7fd0d80038e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:41.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.928+0000 7fd0e77fe700 1 --2- 172.21.15.88:0/1338414154 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0d8003e10 0x7fd0d8004690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:41.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.928+0000 7fd0e77fe700 1 --2- 172.21.15.88:0/1338414154 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd0d8004b80 0x7fd0d8005000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:41.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.928+0000 7fd0e77fe700 1 -- 172.21.15.88:0/1338414154 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f00a7190 con 0x7fd0d8003e10 2023-10-18T16:07:41.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.928+0000 7fd0e77fe700 1 -- 172.21.15.88:0/1338414154 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f00c3e10 con 0x7fd0d8004b80 2023-10-18T16:07:41.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.928+0000 7fd0e77fe700 1 -- 172.21.15.88:0/1338414154 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d800b8e0 con 0x7fd0f01508a0 2023-10-18T16:07:41.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.928+0000 7fd0e7fff700 1 --2- 172.21.15.88:0/1338414154 >> v2:172.21.15.151:3300/0 conn(0x7fd0f00c2e70 0x7fd0f01503b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2023-10-18T16:07:41.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.928+0000 7fd0f4d71700 1 --2- 172.21.15.88:0/1338414154 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd0f01508a0 0x7fd0d80038e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:41.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.928+0000 7fd0e7fff700 1 --2- 172.21.15.88:0/1338414154 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0d8003e10 0x7fd0d8004690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:41.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.929+0000 7fd0f5572700 1 --2- 172.21.15.88:0/1338414154 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd0d8004b80 0x7fd0d8005000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:41.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.929+0000 7fd0e7fff700 1 -- 172.21.15.88:0/1338414154 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd0f01508a0 msgr2=0x7fd0d80038e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:41.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.929+0000 7fd0e7fff700 1 --2- 172.21.15.88:0/1338414154 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd0f01508a0 0x7fd0d80038e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:41.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.929+0000 7fd0e7fff700 1 -- 172.21.15.88:0/1338414154 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd0d8004b80 msgr2=0x7fd0d8005000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:41.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.929+0000 7fd0e7fff700 1 --2- 172.21.15.88:0/1338414154 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd0d8004b80 0x7fd0d8005000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:41.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.929+0000 7fd0e7fff700 1 -- 172.21.15.88:0/1338414154 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0d80033f0 con 0x7fd0d8003e10 2023-10-18T16:07:41.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.929+0000 7fd0e7fff700 1 --2- 172.21.15.88:0/1338414154 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0d8003e10 0x7fd0d8004690 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fd0e0009c60 tx=0x7fd0e0009060).ready entity=mon.0 client_cookie=1a800405ab109892 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:41.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.929+0000 7fd0e77fe700 1 -- 172.21.15.88:0/1338414154 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0e00095c0 con 0x7fd0d8003e10 2023-10-18T16:07:41.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.929+0000 7fd0e77fe700 1 -- 172.21.15.88:0/1338414154 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0e000e710 con 0x7fd0d8003e10 2023-10-18T16:07:41.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.929+0000 7fd0e77fe700 1 -- 172.21.15.88:0/1338414154 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0e000ea90 con 0x7fd0d8003e10 2023-10-18T16:07:41.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.929+0000 7fd0f6fd5700 1 -- 172.21.15.88:0/1338414154 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0d8003e10 msgr2=0x7fd0d8004690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:41.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.929+0000 7fd0f6fd5700 1 --2- 172.21.15.88:0/1338414154 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0d8003e10 0x7fd0d8004690 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fd0e0009c60 tx=0x7fd0e0009060).stop 2023-10-18T16:07:41.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.929+0000 7fd0f5572700 1 -- 172.21.15.88:0/1338414154 reap_dead start 2023-10-18T16:07:41.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.930+0000 7fd0f6fd5700 1 -- 172.21.15.88:0/1338414154 shutdown_connections 2023-10-18T16:07:41.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.930+0000 7fd0f6fd5700 1 -- 172.21.15.88:0/1338414154 >> 172.21.15.88:0/1338414154 conn(0x7fd0f0069d80 msgr2=0x7fd0f006a190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:41.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.930+0000 7fd0f6fd5700 1 -- 172.21.15.88:0/1338414154 shutdown_connections 2023-10-18T16:07:41.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.930+0000 7fd0f6fd5700 1 -- 172.21.15.88:0/1338414154 wait complete. 2023-10-18T16:07:41.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.931+0000 7fd0f6fd5700 1 Processor -- start 2023-10-18T16:07:41.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.931+0000 7fd0f6fd5700 1 -- start start 2023-10-18T16:07:41.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.931+0000 7fd0f6fd5700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd0d8003e10 0x7fd0f01519d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:41.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.931+0000 7fd0f6fd5700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd0f00c5cc0 0x7fd0f0152880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:41.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.931+0000 7fd0f6fd5700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0f00c6120 0x7fd0f0152d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:41.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.932+0000 7fd0f6fd5700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f0164660 con 0x7fd0f00c6120 2023-10-18T16:07:41.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.932+0000 7fd0f6fd5700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f0164130 con 0x7fd0f00c5cc0 2023-10-18T16:07:41.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.932+0000 7fd0f6fd5700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f0154200 con 0x7fd0d8003e10 2023-10-18T16:07:41.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.932+0000 7fd0f5572700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0f00c6120 0x7fd0f0152d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:41.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.932+0000 7fd0f5572700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0f00c6120 0x7fd0f0152d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43952/0 (socket says 172.21.15.88:43952) 2023-10-18T16:07:41.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.932+0000 7fd0f5572700 1 -- 172.21.15.88:0/613184349 learned_addr learned my addr 172.21.15.88:0/613184349 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:41.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.932+0000 7fd0e7fff700 1 --2- 172.21.15.88:0/613184349 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd0f00c5cc0 0x7fd0f0152880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:41.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.932+0000 7fd0f4d71700 1 --2- 172.21.15.88:0/613184349 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd0d8003e10 0x7fd0f01519d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:41.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.932+0000 7fd0f5572700 1 -- 172.21.15.88:0/613184349 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd0d8003e10 msgr2=0x7fd0f01519d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:41.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.932+0000 7fd0f5572700 1 --2- 172.21.15.88:0/613184349 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd0d8003e10 0x7fd0f01519d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:41.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.932+0000 7fd0f5572700 1 -- 172.21.15.88:0/613184349 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd0f00c5cc0 msgr2=0x7fd0f0152880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:41.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.932+0000 7fd0f5572700 1 --2- 172.21.15.88:0/613184349 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd0f00c5cc0 0x7fd0f0152880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:41.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.932+0000 7fd0f5572700 1 -- 172.21.15.88:0/613184349 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0f0153910 con 0x7fd0f00c6120 2023-10-18T16:07:41.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.932+0000 7fd0f4d71700 1 --2- 172.21.15.88:0/613184349 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd0d8003e10 0x7fd0f01519d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-10-18T16:07:41.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.932+0000 7fd0e7fff700 1 --2- 172.21.15.88:0/613184349 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd0f00c5cc0 0x7fd0f0152880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-10-18T16:07:41.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.932+0000 7fd0f5572700 1 --2- 172.21.15.88:0/613184349 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0f00c6120 0x7fd0f0152d70 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fd0e80094d0 tx=0x7fd0e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:41.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.932+0000 7fd0e4ff9700 1 -- 172.21.15.88:0/613184349 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0e800e040 con 0x7fd0f00c6120 2023-10-18T16:07:41.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.932+0000 7fd0e4ff9700 1 -- 172.21.15.88:0/613184349 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0e800f040 con 0x7fd0f00c6120 2023-10-18T16:07:41.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.932+0000 7fd0f6fd5700 1 -- 172.21.15.88:0/613184349 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0f0164380 con 0x7fd0f00c6120 2023-10-18T16:07:41.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.932+0000 7fd0e4ff9700 1 -- 172.21.15.88:0/613184349 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0e800fe50 con 0x7fd0f00c6120 2023-10-18T16:07:41.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.932+0000 7fd0f6fd5700 1 -- 172.21.15.88:0/613184349 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0f0153e90 con 0x7fd0f00c6120 2023-10-18T16:07:41.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.933+0000 7fd0f6fd5700 1 -- 172.21.15.88:0/613184349 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0f01543b0 con 0x7fd0f00c6120 2023-10-18T16:07:41.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.933+0000 7fd0e4ff9700 1 -- 172.21.15.88:0/613184349 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7fd0e8015660 con 0x7fd0f00c6120 2023-10-18T16:07:41.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.934+0000 7fd0e4ff9700 1 --2- 172.21.15.88:0/613184349 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fd0dc05f180 0x7fd0dc061640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:41.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.934+0000 7fd0e4ff9700 1 -- 172.21.15.88:0/613184349 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6121+0+0 (secure 0 0 0) 0x7fd0e807ad60 con 0x7fd0f00c6120 2023-10-18T16:07:41.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.934+0000 7fd0f4d71700 1 --2- 172.21.15.88:0/613184349 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fd0dc05f180 0x7fd0dc061640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:41.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.934+0000 7fd0f4d71700 1 --2- 172.21.15.88:0/613184349 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fd0dc05f180 0x7fd0dc061640 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7fd0ec001d20 tx=0x7fd0ec009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:41.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:41.938+0000 7fd0e4ff9700 1 -- 172.21.15.88:0/613184349 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7fd0f01543b0 con 0x7fd0f00c6120 2023-10-18T16:07:42.041 INFO:tasks.ceph.mds.d.smithi151.stderr:daemon-helper: command crashed with signal 6 2023-10-18T16:07:42.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.100+0000 7fd0f6fd5700 1 -- 172.21.15.88:0/613184349 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd0f0154370 con 0x7fd0f00c6120 2023-10-18T16:07:42.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.101+0000 7fd0e4ff9700 1 -- 172.21.15.88:0/613184349 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16941 (secure 0 0 0) 0x7fd0e804c030 con 0x7fd0f00c6120 2023-10-18T16:07:42.105 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:07:42.105 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":21,"fsid":"fb1f77e8-ead0-4f90-8128-fdf5137eb16f","created":"2023-10-18T16:07:23.662468+0000","modified":"2023-10-18T16:07:41.782214+0000","last_up_change":"2023-10-18T16:07:32.498203+0000","last_in_change":"2023-10-18T16:07:26.064784+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-10-18T16:07:32.536320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-18T16:07:34.769989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-10-18T16:07:39.439702+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-10-18T16:07:40.753774+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e421c20c-f7bc-4574-97a2-201bb4a16b4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":91007},{"type":"v1","addr":"172.21.15.88:6809","nonce":91007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":91007},{"type":"v1","addr":"172.21.15.88:6811","nonce":91007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":91007},{"type":"v1","addr":"172.21.15.88:6815","nonce":91007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":91007},{"type":"v1","addr":"172.21.15.88:6813","nonce":91007}]},"public_addr":"172.21.15.88:6809/91007","cluster_addr":"172.21.15.88:6811/91007","heartbeat_back_addr":"172.21.15.88:6815/91007","heartbeat_front_addr":"172.21.15.88:6813/91007","state":["exists","up"]},{"osd":1,"uuid":"9485c5ec-379a-4038-a2d8-1f1081fe2eea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":91009},{"type":"v1","addr":"172.21.15.88:6825","nonce":91009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":91009},{"type":"v1","addr":"172.21.15.88:6827","nonce":91009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":91009},{"type":"v1","addr":"172.21.15.88:6831","nonce":91009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":91009},{"type":"v1","addr":"172.21.15.88:6829","nonce":91009}]},"public_addr":"172.21.15.88:6825/91009","cluster_addr":"172.21.15.88:6827/91009","heartbeat_back_addr":"172.21.15.88:6831/91009","heartbeat_front_addr":"172.21.15.88:6829/91009","state":["exists","up"]},{"osd":2,"uuid":"671895e2-57f3-4f1a-ba4a-a580af1f3ec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":91005},{"type":"v1","addr":"172.21.15.88:6801","nonce":91005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":91005},{"type":"v1","addr":"172.21.15.88:6803","nonce":91005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":91005},{"type":"v1","addr":"172.21.15.88:6807","nonce":91005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":91005},{"type":"v1","addr":"172.21.15.88:6805","nonce":91005}]},"public_addr":"172.21.15.88:6801/91005","cluster_addr":"172.21.15.88:6803/91005","heartbeat_back_addr":"172.21.15.88:6807/91005","heartbeat_front_addr":"172.21.15.88:6805/91005","state":["exists","up"]},{"osd":3,"uuid":"0c4e6e7f-7d9c-4e65-beec-fd0ca384f320","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":91008},{"type":"v1","addr":"172.21.15.88:6817","nonce":91008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":91008},{"type":"v1","addr":"172.21.15.88:6819","nonce":91008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":91008},{"type":"v1","addr":"172.21.15.88:6823","nonce":91008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":91008},{"type":"v1","addr":"172.21.15.88:6821","nonce":91008}]},"public_addr":"172.21.15.88:6817/91008","cluster_addr":"172.21.15.88:6819/91008","heartbeat_back_addr":"172.21.15.88:6823/91008","heartbeat_front_addr":"172.21.15.88:6821/91008","state":["exists","up"]},{"osd":4,"uuid":"29570691-ef35-4b80-9247-aa3703877af4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":91514},{"type":"v1","addr":"172.21.15.151:6801","nonce":91514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":91514},{"type":"v1","addr":"172.21.15.151:6803","nonce":91514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":91514},{"type":"v1","addr":"172.21.15.151:6807","nonce":91514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":91514},{"type":"v1","addr":"172.21.15.151:6805","nonce":91514}]},"public_addr":"172.21.15.151:6801/91514","cluster_addr":"172.21.15.151:6803/91514","heartbeat_back_addr":"172.21.15.151:6807/91514","heartbeat_front_addr":"172.21.15.151:6805/91514","state":["exists","up"]},{"osd":5,"uuid":"26540b7b-032a-4b3b-bf6c-b131a449fa99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":91513},{"type":"v1","addr":"172.21.15.151:6809","nonce":91513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":91513},{"type":"v1","addr":"172.21.15.151:6811","nonce":91513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":91513},{"type":"v1","addr":"172.21.15.151:6815","nonce":91513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":91513},{"type":"v1","addr":"172.21.15.151:6813","nonce":91513}]},"public_addr":"172.21.15.151:6809/91513","cluster_addr":"172.21.15.151:6811/91513","heartbeat_back_addr":"172.21.15.151:6815/91513","heartbeat_front_addr":"172.21.15.151:6813/91513","state":["exists","up"]},{"osd":6,"uuid":"7af01f9c-90db-45fd-93c8-09517c9b0881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":91544},{"type":"v1","addr":"172.21.15.151:6817","nonce":91544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":91544},{"type":"v1","addr":"172.21.15.151:6819","nonce":91544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":91544},{"type":"v1","addr":"172.21.15.151:6823","nonce":91544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":91544},{"type":"v1","addr":"172.21.15.151:6821","nonce":91544}]},"public_addr":"172.21.15.151:6817/91544","cluster_addr":"172.21.15.151:6819/91544","heartbeat_back_addr":"172.21.15.151:6823/91544","heartbeat_front_addr":"172.21.15.151:6821/91544","state":["exists","up"]},{"osd":7,"uuid":"b527b0a5-4249-4854-a30f-bff897757239","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":91545},{"type":"v1","addr":"172.21.15.151:6825","nonce":91545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":91545},{"type":"v1","addr":"172.21.15.151:6827","nonce":91545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":91545},{"type":"v1","addr":"172.21.15.151:6831","nonce":91545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":91545},{"type":"v1","addr":"172.21.15.151:6829","nonce":91545}]},"public_addr":"172.21.15.151:6825/91545","cluster_addr":"172.21.15.151:6827/91545","heartbeat_back_addr":"172.21.15.151:6831/91545","heartbeat_front_addr":"172.21.15.151:6829/91545","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T16:07:31.216742+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T16:07:31.467229+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T16:07:31.220809+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T16:07:31.414999+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T16:07:31.188976+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T16:07:31.252153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T16:07:31.254324+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T16:07:31.203434+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-18T16:07:42.106 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.103+0000 7fd0f6fd5700 1 -- 172.21.15.88:0/613184349 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fd0dc05f180 msgr2=0x7fd0dc061640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:42.107 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.103+0000 7fd0f6fd5700 1 --2- 172.21.15.88:0/613184349 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fd0dc05f180 0x7fd0dc061640 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7fd0ec001d20 tx=0x7fd0ec009040).stop 2023-10-18T16:07:42.107 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.103+0000 7fd0f6fd5700 1 -- 172.21.15.88:0/613184349 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0f00c6120 msgr2=0x7fd0f0152d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:42.107 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.103+0000 7fd0f6fd5700 1 --2- 172.21.15.88:0/613184349 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0f00c6120 0x7fd0f0152d70 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fd0e80094d0 tx=0x7fd0e800b040).stop 2023-10-18T16:07:42.107 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.103+0000 7fd0f6fd5700 1 -- 172.21.15.88:0/613184349 shutdown_connections 2023-10-18T16:07:42.107 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.103+0000 7fd0f6fd5700 1 --2- 172.21.15.88:0/613184349 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fd0dc05f180 0x7fd0dc061640 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:42.108 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.103+0000 7fd0f6fd5700 1 --2- 172.21.15.88:0/613184349 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0f00c6120 0x7fd0f0152d70 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:42.108 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.103+0000 7fd0f6fd5700 1 --2- 172.21.15.88:0/613184349 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd0d8003e10 0x7fd0f01519d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:42.108 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.103+0000 7fd0f6fd5700 1 --2- 172.21.15.88:0/613184349 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd0f00c5cc0 0x7fd0f0152880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:42.108 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.103+0000 7fd0f6fd5700 1 -- 172.21.15.88:0/613184349 >> 172.21.15.88:0/613184349 conn(0x7fd0f0069d80 msgr2=0x7fd0f014eef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:42.108 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.103+0000 7fd0f6fd5700 1 -- 172.21.15.88:0/613184349 shutdown_connections 2023-10-18T16:07:42.109 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.103+0000 7fd0f6fd5700 1 -- 172.21.15.88:0/613184349 wait complete. 2023-10-18T16:07:42.118 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-18T16:07:42.232 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.228+0000 7f0c05307700 1 Processor -- start 2023-10-18T16:07:42.238 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.228+0000 7f0c05307700 1 -- start start 2023-10-18T16:07:42.244 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.229+0000 7f0c05307700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0c0014d7c0 0x7f0c000a8de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:42.244 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.229+0000 7f0c05307700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f0c00064360 0x7f0c001538a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:42.244 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.229+0000 7f0c05307700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c00151840 con 0x7f0c0014dba0 2023-10-18T16:07:42.244 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.229+0000 7f0c05307700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c000c42d0 con 0x7f0c00064360 2023-10-18T16:07:42.245 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.229+0000 7f0c05307700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c00064830 con 0x7f0c0014d7c0 2023-10-18T16:07:42.245 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.229+0000 7f0c04b06700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f0c0014dba0 0x7f0c00150350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:52368/0 (socket says 172.21.15.88:52368) 2023-10-18T16:07:42.245 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.229+0000 7f0c04b06700 1 -- 172.21.15.88:0/468016345 learned_addr learned my addr 172.21.15.88:0/468016345 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:07:42.245 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.229+0000 7f0bff7fe700 1 --2- 172.21.15.88:0/468016345 >> v2:172.21.15.151:3300/0 conn(0x7f0c00064360 0x7f0c001538a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:42.246 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.229+0000 7f0bfffff700 1 --2- 172.21.15.88:0/468016345 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0c0014d7c0 0x7f0c000a8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:42.246 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.229+0000 7f0bfeffd700 1 -- 172.21.15.88:0/468016345 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2868409207 0 0) 0x7f0c00151840 con 0x7f0c0014dba0 2023-10-18T16:07:42.246 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.229+0000 7f0bfeffd700 1 -- 172.21.15.88:0/468016345 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0be80033f0 con 0x7f0c0014dba0 2023-10-18T16:07:42.246 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.230+0000 7f0bfeffd700 1 -- 172.21.15.88:0/468016345 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3598759252 0 0) 0x7f0be80033f0 con 0x7f0c0014dba0 2023-10-18T16:07:42.246 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.230+0000 7f0bfeffd700 1 -- 172.21.15.88:0/468016345 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0c0014d7c0 msgr2=0x7f0c000a8de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:42.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.230+0000 7f0bfeffd700 1 --2- 172.21.15.88:0/468016345 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0c0014d7c0 0x7f0c000a8de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:42.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.230+0000 7f0bfeffd700 1 -- 172.21.15.88:0/468016345 >> v2:172.21.15.151:3300/0 conn(0x7f0c00064360 msgr2=0x7f0c001538a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:42.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.230+0000 7f0bfeffd700 1 --2- 172.21.15.88:0/468016345 >> v2:172.21.15.151:3300/0 conn(0x7f0c00064360 0x7f0c001538a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:42.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.230+0000 7f0bfeffd700 1 -- 172.21.15.88:0/468016345 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c00064b10 con 0x7f0c0014dba0 2023-10-18T16:07:42.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.230+0000 7f0bfeffd700 1 -- 172.21.15.88:0/468016345 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441715075 0 0) 0x7f0bf80017f0 con 0x7f0c0014dba0 2023-10-18T16:07:42.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.230+0000 7f0bfeffd700 1 -- 172.21.15.88:0/468016345 >> v1:172.21.15.88:6789/0 conn(0x7f0c0014dba0 legacy=0x7f0c00150350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:42.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.230+0000 7f0bfeffd700 1 --2- 172.21.15.88:0/468016345 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0c0014d7c0 0x7f0be80038e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:42.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.230+0000 7f0bfeffd700 1 --2- 172.21.15.88:0/468016345 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0be8003e10 0x7f0be8004690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:42.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.230+0000 7f0bfeffd700 1 --2- 172.21.15.88:0/468016345 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0be8004b80 0x7f0be8005000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:42.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.230+0000 7f0bfeffd700 1 -- 172.21.15.88:0/468016345 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c000c42d0 con 0x7f0be8003e10 2023-10-18T16:07:42.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.230+0000 7f0bfeffd700 1 -- 172.21.15.88:0/468016345 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c00064830 con 0x7f0be8004b80 2023-10-18T16:07:42.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.230+0000 7f0bfeffd700 1 -- 172.21.15.88:0/468016345 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0be800b8e0 con 0x7f0c0014d7c0 2023-10-18T16:07:42.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.230+0000 7f0bff7fe700 1 --2- 172.21.15.88:0/468016345 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0be8003e10 0x7f0be8004690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:42.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.230+0000 7f0bfffff700 1 --2- 172.21.15.88:0/468016345 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0c0014d7c0 0x7f0be80038e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:42.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.230+0000 7f0c04b06700 1 --2- 172.21.15.88:0/468016345 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0be8004b80 0x7f0be8005000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:42.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.230+0000 7f0bff7fe700 1 -- 172.21.15.88:0/468016345 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0c0014d7c0 msgr2=0x7f0be80038e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:42.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.230+0000 7f0bff7fe700 1 --2- 172.21.15.88:0/468016345 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0c0014d7c0 0x7f0be80038e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:42.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.230+0000 7f0bff7fe700 1 -- 172.21.15.88:0/468016345 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0be8004b80 msgr2=0x7f0be8005000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:42.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.230+0000 7f0bff7fe700 1 --2- 172.21.15.88:0/468016345 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0be8004b80 0x7f0be8005000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:42.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.230+0000 7f0bff7fe700 1 -- 172.21.15.88:0/468016345 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0be80033f0 con 0x7f0be8003e10 2023-10-18T16:07:42.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.230+0000 7f0bff7fe700 1 --2- 172.21.15.88:0/468016345 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0be8003e10 0x7f0be8004690 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f0c000bd9d0 tx=0x7f0bf4009060).ready entity=mon.0 client_cookie=194d200447adea68 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:42.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.230+0000 7f0bfeffd700 1 -- 172.21.15.88:0/468016345 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf4009e90 con 0x7f0be8003e10 2023-10-18T16:07:42.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.230+0000 7f0bfeffd700 1 -- 172.21.15.88:0/468016345 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bf4009430 con 0x7f0be8003e10 2023-10-18T16:07:42.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.230+0000 7f0bfeffd700 1 -- 172.21.15.88:0/468016345 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf400e850 con 0x7f0be8003e10 2023-10-18T16:07:42.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.231+0000 7f0c05307700 1 -- 172.21.15.88:0/468016345 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0be8003e10 msgr2=0x7f0be8004690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:42.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.231+0000 7f0c05307700 1 --2- 172.21.15.88:0/468016345 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0be8003e10 0x7f0be8004690 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f0c000bd9d0 tx=0x7f0bf4009060).stop 2023-10-18T16:07:42.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.231+0000 7f0c04b06700 1 -- 172.21.15.88:0/468016345 reap_dead start 2023-10-18T16:07:42.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.231+0000 7f0c05307700 1 -- 172.21.15.88:0/468016345 shutdown_connections 2023-10-18T16:07:42.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.231+0000 7f0c05307700 1 -- 172.21.15.88:0/468016345 >> 172.21.15.88:0/468016345 conn(0x7f0c00069d80 msgr2=0x7f0c0006a190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:42.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.231+0000 7f0c05307700 1 -- 172.21.15.88:0/468016345 shutdown_connections 2023-10-18T16:07:42.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.231+0000 7f0c05307700 1 -- 172.21.15.88:0/468016345 wait complete. 2023-10-18T16:07:42.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.232+0000 7f0c05307700 1 Processor -- start 2023-10-18T16:07:42.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.232+0000 7f0c05307700 1 -- start start 2023-10-18T16:07:42.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.233+0000 7f0c05307700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0be8003e10 0x7f0c0014e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:42.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.233+0000 7f0c05307700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0c0014e690 0x7f0c0014fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:42.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.233+0000 7f0c05307700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0c001512a0 0x7f0c001504e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:42.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.233+0000 7f0c05307700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0014f670 con 0x7f0c001512a0 2023-10-18T16:07:42.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.233+0000 7f0c05307700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c001509d0 con 0x7f0c0014e690 2023-10-18T16:07:42.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.233+0000 7f0c05307700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c000bbef0 con 0x7f0be8003e10 2023-10-18T16:07:42.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.233+0000 7f0c04b06700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0c001512a0 0x7f0c001504e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:42.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.233+0000 7f0c04b06700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0c001512a0 0x7f0c001504e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43956/0 (socket says 172.21.15.88:43956) 2023-10-18T16:07:42.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.233+0000 7f0c04b06700 1 -- 172.21.15.88:0/3436432562 learned_addr learned my addr 172.21.15.88:0/3436432562 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:42.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.233+0000 7f0bfffff700 1 --2- 172.21.15.88:0/3436432562 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0be8003e10 0x7f0c0014e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:42.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.233+0000 7f0bff7fe700 1 --2- 172.21.15.88:0/3436432562 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0c0014e690 0x7f0c0014fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:42.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.233+0000 7f0c04b06700 1 -- 172.21.15.88:0/3436432562 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0be8003e10 msgr2=0x7f0c0014e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:42.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.233+0000 7f0c04b06700 1 --2- 172.21.15.88:0/3436432562 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0be8003e10 0x7f0c0014e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:42.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.233+0000 7f0c04b06700 1 -- 172.21.15.88:0/3436432562 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0c0014e690 msgr2=0x7f0c0014fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:42.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.233+0000 7f0c04b06700 1 --2- 172.21.15.88:0/3436432562 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0c0014e690 0x7f0c0014fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:42.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.233+0000 7f0c04b06700 1 -- 172.21.15.88:0/3436432562 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c000bd3f0 con 0x7f0c001512a0 2023-10-18T16:07:42.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.233+0000 7f0c04b06700 1 --2- 172.21.15.88:0/3436432562 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0c001512a0 0x7f0c001504e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f0bf8000f80 tx=0x7f0c000a8330).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:42.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.233+0000 7f0be7fff700 1 -- 172.21.15.88:0/3436432562 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf800f8a0 con 0x7f0c001512a0 2023-10-18T16:07:42.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.234+0000 7f0be7fff700 1 -- 172.21.15.88:0/3436432562 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bf800fa20 con 0x7f0c001512a0 2023-10-18T16:07:42.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.234+0000 7f0c05307700 1 -- 172.21.15.88:0/3436432562 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c000bd6b0 con 0x7f0c001512a0 2023-10-18T16:07:42.258 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.234+0000 7f0be7fff700 1 -- 172.21.15.88:0/3436432562 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf800fde0 con 0x7f0c001512a0 2023-10-18T16:07:42.258 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.234+0000 7f0c05307700 1 -- 172.21.15.88:0/3436432562 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c000bc470 con 0x7f0c001512a0 2023-10-18T16:07:42.258 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.235+0000 7f0c05307700 1 -- 172.21.15.88:0/3436432562 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c00083b10 con 0x7f0c001512a0 2023-10-18T16:07:42.258 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.235+0000 7f0be7fff700 1 -- 172.21.15.88:0/3436432562 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f0bf8012560 con 0x7f0c001512a0 2023-10-18T16:07:42.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.235+0000 7f0be7fff700 1 --2- 172.21.15.88:0/3436432562 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f0bec05f180 0x7f0bec061640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:42.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.235+0000 7f0be7fff700 1 -- 172.21.15.88:0/3436432562 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6121+0+0 (secure 0 0 0) 0x7f0bf8079bd0 con 0x7f0c001512a0 2023-10-18T16:07:42.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.235+0000 7f0bfffff700 1 --2- 172.21.15.88:0/3436432562 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f0bec05f180 0x7f0bec061640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:42.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.236+0000 7f0bfffff700 1 --2- 172.21.15.88:0/3436432562 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f0bec05f180 0x7f0bec061640 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f0bf0006d80 tx=0x7f0bf0009060).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:42.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.239+0000 7f0be7fff700 1 -- 172.21.15.88:0/3436432562 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f0bf804aea0 con 0x7f0c001512a0 2023-10-18T16:07:42.405 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.402+0000 7f0c05307700 1 -- 172.21.15.88:0/3436432562 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0c00064df0 con 0x7f0c001512a0 2023-10-18T16:07:42.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.403+0000 7f0be7fff700 1 -- 172.21.15.88:0/3436432562 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16941 (secure 0 0 0) 0x7f0bf8019030 con 0x7f0c001512a0 2023-10-18T16:07:42.407 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:07:42.407 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":21,"fsid":"fb1f77e8-ead0-4f90-8128-fdf5137eb16f","created":"2023-10-18T16:07:23.662468+0000","modified":"2023-10-18T16:07:41.782214+0000","last_up_change":"2023-10-18T16:07:32.498203+0000","last_in_change":"2023-10-18T16:07:26.064784+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-10-18T16:07:32.536320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-18T16:07:34.769989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-10-18T16:07:39.439702+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-10-18T16:07:40.753774+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e421c20c-f7bc-4574-97a2-201bb4a16b4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":91007},{"type":"v1","addr":"172.21.15.88:6809","nonce":91007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":91007},{"type":"v1","addr":"172.21.15.88:6811","nonce":91007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":91007},{"type":"v1","addr":"172.21.15.88:6815","nonce":91007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":91007},{"type":"v1","addr":"172.21.15.88:6813","nonce":91007}]},"public_addr":"172.21.15.88:6809/91007","cluster_addr":"172.21.15.88:6811/91007","heartbeat_back_addr":"172.21.15.88:6815/91007","heartbeat_front_addr":"172.21.15.88:6813/91007","state":["exists","up"]},{"osd":1,"uuid":"9485c5ec-379a-4038-a2d8-1f1081fe2eea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":91009},{"type":"v1","addr":"172.21.15.88:6825","nonce":91009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":91009},{"type":"v1","addr":"172.21.15.88:6827","nonce":91009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":91009},{"type":"v1","addr":"172.21.15.88:6831","nonce":91009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":91009},{"type":"v1","addr":"172.21.15.88:6829","nonce":91009}]},"public_addr":"172.21.15.88:6825/91009","cluster_addr":"172.21.15.88:6827/91009","heartbeat_back_addr":"172.21.15.88:6831/91009","heartbeat_front_addr":"172.21.15.88:6829/91009","state":["exists","up"]},{"osd":2,"uuid":"671895e2-57f3-4f1a-ba4a-a580af1f3ec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":91005},{"type":"v1","addr":"172.21.15.88:6801","nonce":91005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":91005},{"type":"v1","addr":"172.21.15.88:6803","nonce":91005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":91005},{"type":"v1","addr":"172.21.15.88:6807","nonce":91005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":91005},{"type":"v1","addr":"172.21.15.88:6805","nonce":91005}]},"public_addr":"172.21.15.88:6801/91005","cluster_addr":"172.21.15.88:6803/91005","heartbeat_back_addr":"172.21.15.88:6807/91005","heartbeat_front_addr":"172.21.15.88:6805/91005","state":["exists","up"]},{"osd":3,"uuid":"0c4e6e7f-7d9c-4e65-beec-fd0ca384f320","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":91008},{"type":"v1","addr":"172.21.15.88:6817","nonce":91008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":91008},{"type":"v1","addr":"172.21.15.88:6819","nonce":91008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":91008},{"type":"v1","addr":"172.21.15.88:6823","nonce":91008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":91008},{"type":"v1","addr":"172.21.15.88:6821","nonce":91008}]},"public_addr":"172.21.15.88:6817/91008","cluster_addr":"172.21.15.88:6819/91008","heartbeat_back_addr":"172.21.15.88:6823/91008","heartbeat_front_addr":"172.21.15.88:6821/91008","state":["exists","up"]},{"osd":4,"uuid":"29570691-ef35-4b80-9247-aa3703877af4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":91514},{"type":"v1","addr":"172.21.15.151:6801","nonce":91514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":91514},{"type":"v1","addr":"172.21.15.151:6803","nonce":91514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":91514},{"type":"v1","addr":"172.21.15.151:6807","nonce":91514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":91514},{"type":"v1","addr":"172.21.15.151:6805","nonce":91514}]},"public_addr":"172.21.15.151:6801/91514","cluster_addr":"172.21.15.151:6803/91514","heartbeat_back_addr":"172.21.15.151:6807/91514","heartbeat_front_addr":"172.21.15.151:6805/91514","state":["exists","up"]},{"osd":5,"uuid":"26540b7b-032a-4b3b-bf6c-b131a449fa99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":91513},{"type":"v1","addr":"172.21.15.151:6809","nonce":91513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":91513},{"type":"v1","addr":"172.21.15.151:6811","nonce":91513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":91513},{"type":"v1","addr":"172.21.15.151:6815","nonce":91513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":91513},{"type":"v1","addr":"172.21.15.151:6813","nonce":91513}]},"public_addr":"172.21.15.151:6809/91513","cluster_addr":"172.21.15.151:6811/91513","heartbeat_back_addr":"172.21.15.151:6815/91513","heartbeat_front_addr":"172.21.15.151:6813/91513","state":["exists","up"]},{"osd":6,"uuid":"7af01f9c-90db-45fd-93c8-09517c9b0881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":91544},{"type":"v1","addr":"172.21.15.151:6817","nonce":91544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":91544},{"type":"v1","addr":"172.21.15.151:6819","nonce":91544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":91544},{"type":"v1","addr":"172.21.15.151:6823","nonce":91544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":91544},{"type":"v1","addr":"172.21.15.151:6821","nonce":91544}]},"public_addr":"172.21.15.151:6817/91544","cluster_addr":"172.21.15.151:6819/91544","heartbeat_back_addr":"172.21.15.151:6823/91544","heartbeat_front_addr":"172.21.15.151:6821/91544","state":["exists","up"]},{"osd":7,"uuid":"b527b0a5-4249-4854-a30f-bff897757239","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":91545},{"type":"v1","addr":"172.21.15.151:6825","nonce":91545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":91545},{"type":"v1","addr":"172.21.15.151:6827","nonce":91545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":91545},{"type":"v1","addr":"172.21.15.151:6831","nonce":91545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":91545},{"type":"v1","addr":"172.21.15.151:6829","nonce":91545}]},"public_addr":"172.21.15.151:6825/91545","cluster_addr":"172.21.15.151:6827/91545","heartbeat_back_addr":"172.21.15.151:6831/91545","heartbeat_front_addr":"172.21.15.151:6829/91545","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T16:07:31.216742+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T16:07:31.467229+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T16:07:31.220809+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T16:07:31.414999+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T16:07:31.188976+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T16:07:31.252153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T16:07:31.254324+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T16:07:31.203434+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-18T16:07:42.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.404+0000 7f0c05307700 1 -- 172.21.15.88:0/3436432562 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f0bec05f180 msgr2=0x7f0bec061640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:42.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.404+0000 7f0c05307700 1 --2- 172.21.15.88:0/3436432562 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f0bec05f180 0x7f0bec061640 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f0bf0006d80 tx=0x7f0bf0009060).stop 2023-10-18T16:07:42.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.405+0000 7f0c05307700 1 -- 172.21.15.88:0/3436432562 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0c001512a0 msgr2=0x7f0c001504e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:42.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.405+0000 7f0c05307700 1 --2- 172.21.15.88:0/3436432562 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0c001512a0 0x7f0c001504e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f0bf8000f80 tx=0x7f0c000a8330).stop 2023-10-18T16:07:42.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.405+0000 7f0c05307700 1 -- 172.21.15.88:0/3436432562 shutdown_connections 2023-10-18T16:07:42.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.405+0000 7f0c05307700 1 --2- 172.21.15.88:0/3436432562 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f0bec05f180 0x7f0bec061640 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:42.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.405+0000 7f0c05307700 1 --2- 172.21.15.88:0/3436432562 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0c001512a0 0x7f0c001504e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:42.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.405+0000 7f0c05307700 1 --2- 172.21.15.88:0/3436432562 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0be8003e10 0x7f0c0014e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:42.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.405+0000 7f0c05307700 1 --2- 172.21.15.88:0/3436432562 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0c0014e690 0x7f0c0014fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:42.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.405+0000 7f0c05307700 1 -- 172.21.15.88:0/3436432562 >> 172.21.15.88:0/3436432562 conn(0x7f0c00069d80 msgr2=0x7f0c000c0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:42.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.405+0000 7f0c05307700 1 -- 172.21.15.88:0/3436432562 shutdown_connections 2023-10-18T16:07:42.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.405+0000 7f0c05307700 1 -- 172.21.15.88:0/3436432562 wait complete. 2023-10-18T16:07:42.420 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-18T16:07:42.537 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.533+0000 7fe6166a6700 1 Processor -- start 2023-10-18T16:07:42.537 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.533+0000 7fe6166a6700 1 -- start start 2023-10-18T16:07:42.540 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.534+0000 7fe6166a6700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe61014db20 0x7fe610154430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:42.540 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.534+0000 7fe6166a6700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe6100c22e0 0x7fe6100a8650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:42.540 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.534+0000 7fe6166a6700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fe6100c2760 0x7fe6100c34e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:42.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.534+0000 7fe6166a6700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6100c31e0 con 0x7fe6100c22e0 2023-10-18T16:07:42.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.534+0000 7fe6166a6700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6100c1e50 con 0x7fe6100c2760 2023-10-18T16:07:42.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.534+0000 7fe6166a6700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6100a90b0 con 0x7fe61014db20 2023-10-18T16:07:42.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.534+0000 7fe60f7fe700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe6100c22e0 0x7fe6100a8650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:42.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.534+0000 7fe60f7fe700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe6100c22e0 0x7fe6100a8650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43964/0 (socket says 172.21.15.88:43964) 2023-10-18T16:07:42.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.534+0000 7fe60f7fe700 1 -- 172.21.15.88:0/187306502 learned_addr learned my addr 172.21.15.88:0/187306502 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:42.542 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.534+0000 7fe60ffff700 1 --2- 172.21.15.88:0/187306502 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe61014db20 0x7fe610154430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:42.542 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.534+0000 7fe614c43700 1 --2- 172.21.15.88:0/187306502 >> v2:172.21.15.151:3300/0 conn(0x7fe6100c2760 0x7fe6100c34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:42.542 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.534+0000 7fe60f7fe700 1 -- 172.21.15.88:0/187306502 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe61014db20 msgr2=0x7fe610154430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:42.542 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.534+0000 7fe60f7fe700 1 --2- 172.21.15.88:0/187306502 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe61014db20 0x7fe610154430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:42.542 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.534+0000 7fe60f7fe700 1 -- 172.21.15.88:0/187306502 >> v2:172.21.15.151:3300/0 conn(0x7fe6100c2760 msgr2=0x7fe6100c34e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:42.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.534+0000 7fe60f7fe700 1 --2- 172.21.15.88:0/187306502 >> v2:172.21.15.151:3300/0 conn(0x7fe6100c2760 0x7fe6100c34e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:42.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.534+0000 7fe60f7fe700 1 -- 172.21.15.88:0/187306502 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6100a9390 con 0x7fe6100c22e0 2023-10-18T16:07:42.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.534+0000 7fe60ffff700 1 --2- 172.21.15.88:0/187306502 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe61014db20 0x7fe610154430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:07:42.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.534+0000 7fe614c43700 1 --2- 172.21.15.88:0/187306502 >> v2:172.21.15.151:3300/0 conn(0x7fe6100c2760 0x7fe6100c34e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:07:42.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.535+0000 7fe60f7fe700 1 --2- 172.21.15.88:0/187306502 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe6100c22e0 0x7fe6100a8650 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fe610083b20 tx=0x7fe6040090f0).ready entity=mon.0 client_cookie=cafb309fb6810d4d server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:42.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.535+0000 7fe60effd700 1 -- 172.21.15.88:0/187306502 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe60400b600 con 0x7fe6100c22e0 2023-10-18T16:07:42.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.535+0000 7fe60effd700 1 -- 172.21.15.88:0/187306502 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe60400b760 con 0x7fe6100c22e0 2023-10-18T16:07:42.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.535+0000 7fe60effd700 1 -- 172.21.15.88:0/187306502 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe60400bae0 con 0x7fe6100c22e0 2023-10-18T16:07:42.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.535+0000 7fe6166a6700 1 -- 172.21.15.88:0/187306502 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe6100c22e0 msgr2=0x7fe6100a8650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:42.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.535+0000 7fe6166a6700 1 --2- 172.21.15.88:0/187306502 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe6100c22e0 0x7fe6100a8650 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fe610083b20 tx=0x7fe6040090f0).stop 2023-10-18T16:07:42.545 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.535+0000 7fe6166a6700 1 -- 172.21.15.88:0/187306502 shutdown_connections 2023-10-18T16:07:42.545 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.535+0000 7fe6166a6700 1 --2- 172.21.15.88:0/187306502 >> v2:172.21.15.151:3300/0 conn(0x7fe6100c2760 0x7fe6100c34e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:42.545 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.535+0000 7fe6166a6700 1 --2- 172.21.15.88:0/187306502 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe61014db20 0x7fe610154430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:42.545 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.535+0000 7fe6166a6700 1 --2- 172.21.15.88:0/187306502 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe6100c22e0 0x7fe6100a8650 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:42.545 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.535+0000 7fe6166a6700 1 -- 172.21.15.88:0/187306502 >> 172.21.15.88:0/187306502 conn(0x7fe610069d80 msgr2=0x7fe61006a190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:42.545 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.535+0000 7fe6166a6700 1 -- 172.21.15.88:0/187306502 shutdown_connections 2023-10-18T16:07:42.546 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.535+0000 7fe6166a6700 1 -- 172.21.15.88:0/187306502 wait complete. 2023-10-18T16:07:42.546 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.536+0000 7fe6166a6700 1 Processor -- start 2023-10-18T16:07:42.546 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.537+0000 7fe6166a6700 1 -- start start 2023-10-18T16:07:42.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.537+0000 7fe6166a6700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe6100c22e0 0x7fe6100c09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:42.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.537+0000 7fe6166a6700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe6100c2760 0x7fe61014fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:42.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.537+0000 7fe6166a6700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe61014db20 0x7fe610160280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:42.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.537+0000 7fe6166a6700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6100c0240 con 0x7fe6100c22e0 2023-10-18T16:07:42.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.537+0000 7fe6166a6700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6100acba0 con 0x7fe6100c2760 2023-10-18T16:07:42.550 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.537+0000 7fe6166a6700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6100bfbc0 con 0x7fe61014db20 2023-10-18T16:07:42.550 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.537+0000 7fe60ffff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe6100c22e0 0x7fe6100c09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:42.550 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.537+0000 7fe60ffff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe6100c22e0 0x7fe6100c09b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43968/0 (socket says 172.21.15.88:43968) 2023-10-18T16:07:42.550 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.537+0000 7fe60ffff700 1 -- 172.21.15.88:0/4230683338 learned_addr learned my addr 172.21.15.88:0/4230683338 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:42.550 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.538+0000 7fe614c43700 1 --2- 172.21.15.88:0/4230683338 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe61014db20 0x7fe610160280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:42.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.538+0000 7fe60f7fe700 1 --2- 172.21.15.88:0/4230683338 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe6100c2760 0x7fe61014fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:42.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.538+0000 7fe60ffff700 1 -- 172.21.15.88:0/4230683338 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe61014db20 msgr2=0x7fe610160280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:42.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.538+0000 7fe60ffff700 1 --2- 172.21.15.88:0/4230683338 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe61014db20 0x7fe610160280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:42.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.538+0000 7fe60ffff700 1 -- 172.21.15.88:0/4230683338 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe6100c2760 msgr2=0x7fe61014fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:42.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.538+0000 7fe60ffff700 1 --2- 172.21.15.88:0/4230683338 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe6100c2760 0x7fe61014fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:42.552 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.538+0000 7fe60ffff700 1 -- 172.21.15.88:0/4230683338 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe604009370 con 0x7fe6100c22e0 2023-10-18T16:07:42.552 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.538+0000 7fe60ffff700 1 --2- 172.21.15.88:0/4230683338 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe6100c22e0 0x7fe6100c09b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fe60000d890 tx=0x7fe600008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:42.552 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.538+0000 7fe5f7fff700 1 -- 172.21.15.88:0/4230683338 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6000125f0 con 0x7fe6100c22e0 2023-10-18T16:07:42.552 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.538+0000 7fe5f7fff700 1 -- 172.21.15.88:0/4230683338 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe600012750 con 0x7fe6100c22e0 2023-10-18T16:07:42.552 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.538+0000 7fe6166a6700 1 -- 172.21.15.88:0/4230683338 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6100c0660 con 0x7fe6100c22e0 2023-10-18T16:07:42.553 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.538+0000 7fe5f7fff700 1 -- 172.21.15.88:0/4230683338 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe600012ad0 con 0x7fe6100c22e0 2023-10-18T16:07:42.553 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.538+0000 7fe6166a6700 1 -- 172.21.15.88:0/4230683338 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6100ac8c0 con 0x7fe6100c22e0 2023-10-18T16:07:42.553 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.539+0000 7fe6166a6700 1 -- 172.21.15.88:0/4230683338 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe61006c4e0 con 0x7fe6100c22e0 2023-10-18T16:07:42.553 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.539+0000 7fe5f7fff700 1 -- 172.21.15.88:0/4230683338 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7fe600013480 con 0x7fe6100c22e0 2023-10-18T16:07:42.553 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.539+0000 7fe5f7fff700 1 --2- 172.21.15.88:0/4230683338 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fe5fc05f150 0x7fe5fc061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:42.553 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.539+0000 7fe5f7fff700 1 -- 172.21.15.88:0/4230683338 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6121+0+0 (secure 0 0 0) 0x7fe60007a330 con 0x7fe6100c22e0 2023-10-18T16:07:42.554 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.540+0000 7fe60f7fe700 1 --2- 172.21.15.88:0/4230683338 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fe5fc05f150 0x7fe5fc061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:42.554 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.540+0000 7fe60f7fe700 1 --2- 172.21.15.88:0/4230683338 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fe5fc05f150 0x7fe5fc061610 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fe610083b20 tx=0x7fe604017000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:42.554 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.543+0000 7fe5f7fff700 1 -- 172.21.15.88:0/4230683338 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7fe60004b5a0 con 0x7fe6100c22e0 2023-10-18T16:07:42.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:42.749+0000 7fe6166a6700 1 -- 172.21.15.88:0/4230683338 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fe6100bf870 con 0x7fe6100c22e0 2023-10-18T16:07:43.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.423+0000 7fe5f7fff700 1 -- 172.21.15.88:0/4230683338 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v5) v1 ==== 118+0+0 (secure 0 0 0) 0x7fe60004ebc0 con 0x7fe6100c22e0 2023-10-18T16:07:43.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.426+0000 7fe6166a6700 1 -- 172.21.15.88:0/4230683338 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fe5fc05f150 msgr2=0x7fe5fc061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:43.430 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.426+0000 7fe6166a6700 1 --2- 172.21.15.88:0/4230683338 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fe5fc05f150 0x7fe5fc061610 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fe610083b20 tx=0x7fe604017000).stop 2023-10-18T16:07:43.430 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.426+0000 7fe6166a6700 1 -- 172.21.15.88:0/4230683338 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe6100c22e0 msgr2=0x7fe6100c09b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:43.430 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.426+0000 7fe6166a6700 1 --2- 172.21.15.88:0/4230683338 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe6100c22e0 0x7fe6100c09b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fe60000d890 tx=0x7fe600008040).stop 2023-10-18T16:07:43.430 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.426+0000 7fe6166a6700 1 -- 172.21.15.88:0/4230683338 shutdown_connections 2023-10-18T16:07:43.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.426+0000 7fe6166a6700 1 --2- 172.21.15.88:0/4230683338 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fe5fc05f150 0x7fe5fc061610 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:43.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.426+0000 7fe6166a6700 1 --2- 172.21.15.88:0/4230683338 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe61014db20 0x7fe610160280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:43.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.426+0000 7fe6166a6700 1 --2- 172.21.15.88:0/4230683338 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe6100c22e0 0x7fe6100c09b0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:43.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.426+0000 7fe6166a6700 1 --2- 172.21.15.88:0/4230683338 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe6100c2760 0x7fe61014fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:43.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.426+0000 7fe6166a6700 1 -- 172.21.15.88:0/4230683338 >> 172.21.15.88:0/4230683338 conn(0x7fe610069d80 msgr2=0x7fe6100c3380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:43.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.426+0000 7fe6166a6700 1 -- 172.21.15.88:0/4230683338 shutdown_connections 2023-10-18T16:07:43.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.426+0000 7fe6166a6700 1 -- 172.21.15.88:0/4230683338 wait complete. 2023-10-18T16:07:43.444 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 3, 'session_timeout': 300} 2023-10-18T16:07:43.445 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2023-10-18T16:07:43.567 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.563+0000 7f2fbea20700 1 Processor -- start 2023-10-18T16:07:43.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.564+0000 7f2fbea20700 1 -- start start 2023-10-18T16:07:43.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.564+0000 7f2fbea20700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f2fb814b560 0x7f2fb8062650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:43.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.564+0000 7f2fbea20700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fb814e3d0 con 0x7f2fb80bcdc0 2023-10-18T16:07:43.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.564+0000 7f2fbea20700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fb81516c0 con 0x7f2fb814b940 2023-10-18T16:07:43.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.564+0000 7f2fbea20700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fb814e130 con 0x7f2fb814b560 2023-10-18T16:07:43.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.564+0000 7f2fbcfbd700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f2fb80bcdc0 0x7f2fb8158a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:52378/0 (socket says 172.21.15.88:52378) 2023-10-18T16:07:43.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.565+0000 7f2fbcfbd700 1 -- 172.21.15.88:0/1792778000 learned_addr learned my addr 172.21.15.88:0/1792778000 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:07:43.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.565+0000 7f2fb7fff700 1 --2- 172.21.15.88:0/1792778000 >> v2:172.21.15.151:3300/0 conn(0x7f2fb814b560 0x7f2fb8062650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:43.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.565+0000 7f2fb6ffd700 1 -- 172.21.15.88:0/1792778000 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2997926328 0 0) 0x7f2fb814e3d0 con 0x7f2fb80bcdc0 2023-10-18T16:07:43.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.565+0000 7f2fb6ffd700 1 -- 172.21.15.88:0/1792778000 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fa00033f0 con 0x7f2fb80bcdc0 2023-10-18T16:07:43.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.565+0000 7f2fb6ffd700 1 -- 172.21.15.88:0/1792778000 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4238797877 0 0) 0x7f2fa00033f0 con 0x7f2fb80bcdc0 2023-10-18T16:07:43.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.565+0000 7f2fb6ffd700 1 -- 172.21.15.88:0/1792778000 >> v2:172.21.15.151:3300/0 conn(0x7f2fb814b560 msgr2=0x7f2fb8062650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:43.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.565+0000 7f2fb6ffd700 1 --2- 172.21.15.88:0/1792778000 >> v2:172.21.15.151:3300/0 conn(0x7f2fb814b560 0x7f2fb8062650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:43.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.565+0000 7f2fb6ffd700 1 -- 172.21.15.88:0/1792778000 >> v1:172.21.15.151:6789/0 conn(0x7f2fb814b940 legacy=0x7f2fb814e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:43.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.565+0000 7f2fb6ffd700 1 -- 172.21.15.88:0/1792778000 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fb80bf350 con 0x7f2fb80bcdc0 2023-10-18T16:07:43.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.565+0000 7f2fb6ffd700 1 -- 172.21.15.88:0/1792778000 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441715075 0 0) 0x7f2fb00017f0 con 0x7f2fb80bcdc0 2023-10-18T16:07:43.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.565+0000 7f2fb6ffd700 1 -- 172.21.15.88:0/1792778000 >> v1:172.21.15.88:6789/0 conn(0x7f2fb80bcdc0 legacy=0x7f2fb8158a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:43.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.565+0000 7f2fb6ffd700 1 --2- 172.21.15.88:0/1792778000 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2fa00038e0 0x7f2fa0003d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:43.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.565+0000 7f2fb6ffd700 1 --2- 172.21.15.88:0/1792778000 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2fa0004690 0x7f2fa0004b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:43.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.565+0000 7f2fb6ffd700 1 --2- 172.21.15.88:0/1792778000 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2fa0005000 0x7f2fa000b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:43.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.565+0000 7f2fb6ffd700 1 -- 172.21.15.88:0/1792778000 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb814e130 con 0x7f2fa00038e0 2023-10-18T16:07:43.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.565+0000 7f2fb6ffd700 1 -- 172.21.15.88:0/1792778000 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa000beb0 con 0x7f2fa0005000 2023-10-18T16:07:43.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.565+0000 7f2fb6ffd700 1 -- 172.21.15.88:0/1792778000 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa000c070 con 0x7f2fa0004690 2023-10-18T16:07:43.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.565+0000 7f2fb7fff700 1 --2- 172.21.15.88:0/1792778000 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2fa00038e0 0x7f2fa0003d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:43.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.565+0000 7f2fbcfbd700 1 --2- 172.21.15.88:0/1792778000 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2fa0005000 0x7f2fa000b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:43.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.565+0000 7f2fb77fe700 1 --2- 172.21.15.88:0/1792778000 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2fa0004690 0x7f2fa0004b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:43.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.565+0000 7f2fb7fff700 1 -- 172.21.15.88:0/1792778000 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2fa0004690 msgr2=0x7f2fa0004b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:43.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.565+0000 7f2fb7fff700 1 --2- 172.21.15.88:0/1792778000 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2fa0004690 0x7f2fa0004b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:43.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.565+0000 7f2fb7fff700 1 -- 172.21.15.88:0/1792778000 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2fa0005000 msgr2=0x7f2fa000b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:43.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.565+0000 7f2fb7fff700 1 --2- 172.21.15.88:0/1792778000 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2fa0005000 0x7f2fa000b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:43.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.565+0000 7f2fb7fff700 1 -- 172.21.15.88:0/1792778000 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fa00033f0 con 0x7f2fa00038e0 2023-10-18T16:07:43.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.565+0000 7f2fbcfbd700 1 --2- 172.21.15.88:0/1792778000 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2fa0005000 0x7f2fa000b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:07:43.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.565+0000 7f2fb77fe700 1 --2- 172.21.15.88:0/1792778000 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2fa0004690 0x7f2fa0004b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-10-18T16:07:43.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.565+0000 7f2fbcfbd700 1 -- 172.21.15.88:0/1792778000 reap_dead start 2023-10-18T16:07:43.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.566+0000 7f2fb7fff700 1 --2- 172.21.15.88:0/1792778000 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2fa00038e0 0x7f2fa0003d00 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f2fa8009a80 tx=0x7f2fa8009060).ready entity=mon.0 client_cookie=138831215682addf server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:43.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.566+0000 7f2fb6ffd700 1 -- 172.21.15.88:0/1792778000 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa8009430 con 0x7f2fa00038e0 2023-10-18T16:07:43.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.566+0000 7f2fb6ffd700 1 -- 172.21.15.88:0/1792778000 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fa800e640 con 0x7f2fa00038e0 2023-10-18T16:07:43.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.566+0000 7f2fb6ffd700 1 -- 172.21.15.88:0/1792778000 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa800e9c0 con 0x7f2fa00038e0 2023-10-18T16:07:43.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.566+0000 7f2fbea20700 1 -- 172.21.15.88:0/1792778000 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2fa00038e0 msgr2=0x7f2fa0003d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:43.579 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.566+0000 7f2fbea20700 1 --2- 172.21.15.88:0/1792778000 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2fa00038e0 0x7f2fa0003d00 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f2fa8009a80 tx=0x7f2fa8009060).stop 2023-10-18T16:07:43.579 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.566+0000 7f2fbea20700 1 -- 172.21.15.88:0/1792778000 shutdown_connections 2023-10-18T16:07:43.579 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.566+0000 7f2fbea20700 1 --2- 172.21.15.88:0/1792778000 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2fa00038e0 0x7f2fa0003d00 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:43.579 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.566+0000 7f2fbea20700 1 -- 172.21.15.88:0/1792778000 >> 172.21.15.88:0/1792778000 conn(0x7f2fb8065cd0 msgr2=0x7f2fb80660e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:43.579 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.566+0000 7f2fbea20700 1 -- 172.21.15.88:0/1792778000 shutdown_connections 2023-10-18T16:07:43.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.566+0000 7f2fbea20700 1 -- 172.21.15.88:0/1792778000 wait complete. 2023-10-18T16:07:43.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.567+0000 7f2fbea20700 1 Processor -- start 2023-10-18T16:07:43.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.567+0000 7f2fbea20700 1 -- start start 2023-10-18T16:07:43.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.568+0000 7f2fbea20700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2fa00038e0 0x7f2fb80c36c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:43.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.568+0000 7f2fbea20700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2fb815c650 0x7f2fb8147ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:43.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.568+0000 7f2fbea20700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2fb815f0e0 0x7f2fb80bc280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:43.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.568+0000 7f2fbea20700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb815c1c0 con 0x7f2fb815c650 2023-10-18T16:07:43.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.568+0000 7f2fbea20700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb815bb90 con 0x7f2fb815f0e0 2023-10-18T16:07:43.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.568+0000 7f2fbea20700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb815f840 con 0x7f2fa00038e0 2023-10-18T16:07:43.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.568+0000 7f2fb77fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2fb815c650 0x7f2fb8147ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:43.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.568+0000 7f2fb77fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2fb815c650 0x7f2fb8147ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43978/0 (socket says 172.21.15.88:43978) 2023-10-18T16:07:43.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.568+0000 7f2fb77fe700 1 -- 172.21.15.88:0/3823670549 learned_addr learned my addr 172.21.15.88:0/3823670549 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:43.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.568+0000 7f2fbcfbd700 1 --2- 172.21.15.88:0/3823670549 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2fb815f0e0 0x7f2fb80bc280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:43.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.569+0000 7f2fb77fe700 1 -- 172.21.15.88:0/3823670549 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2fa00038e0 msgr2=0x7f2fb80c36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:43.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.569+0000 7f2fb77fe700 1 --2- 172.21.15.88:0/3823670549 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2fa00038e0 0x7f2fb80c36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-10-18T16:07:43.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.569+0000 7f2fb77fe700 1 -- 172.21.15.88:0/3823670549 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2fb815f0e0 msgr2=0x7f2fb80bc280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:43.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.569+0000 7f2fb77fe700 1 --2- 172.21.15.88:0/3823670549 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2fb815f0e0 0x7f2fb80bc280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:43.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.569+0000 7f2fb77fe700 1 -- 172.21.15.88:0/3823670549 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fb815fb20 con 0x7f2fb815c650 2023-10-18T16:07:43.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.569+0000 7f2fbcfbd700 1 --2- 172.21.15.88:0/3823670549 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2fb815f0e0 0x7f2fb80bc280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-10-18T16:07:43.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.569+0000 7f2fb77fe700 1 --2- 172.21.15.88:0/3823670549 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2fb815c650 0x7f2fb8147ae0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f2fac001eb0 tx=0x7f2fac009040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:43.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.569+0000 7f2f9ffff700 1 -- 172.21.15.88:0/3823670549 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fac00dea0 con 0x7f2fb815c650 2023-10-18T16:07:43.586 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.569+0000 7f2f9ffff700 1 -- 172.21.15.88:0/3823670549 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fac0104e0 con 0x7f2fb815c650 2023-10-18T16:07:43.586 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.569+0000 7f2fbea20700 1 -- 172.21.15.88:0/3823670549 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fb815fe00 con 0x7f2fb815c650 2023-10-18T16:07:43.586 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.569+0000 7f2f9ffff700 1 -- 172.21.15.88:0/3823670549 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fac010860 con 0x7f2fb815c650 2023-10-18T16:07:43.586 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.569+0000 7f2fbea20700 1 -- 172.21.15.88:0/3823670549 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fb815f540 con 0x7f2fb815c650 2023-10-18T16:07:43.586 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.570+0000 7f2fbea20700 1 -- 172.21.15.88:0/3823670549 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fb815c4a0 con 0x7f2fb815c650 2023-10-18T16:07:43.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.570+0000 7f2f9ffff700 1 -- 172.21.15.88:0/3823670549 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f2fac010c50 con 0x7f2fb815c650 2023-10-18T16:07:43.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.571+0000 7f2f9ffff700 1 --2- 172.21.15.88:0/3823670549 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f2fa405f150 0x7f2fa4061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:43.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.571+0000 7f2f9ffff700 1 -- 172.21.15.88:0/3823670549 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6121+0+0 (secure 0 0 0) 0x7f2fac079a40 con 0x7f2fb815c650 2023-10-18T16:07:43.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.571+0000 7f2fb7fff700 1 --2- 172.21.15.88:0/3823670549 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f2fa405f150 0x7f2fa4061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:43.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.571+0000 7f2fb7fff700 1 --2- 172.21.15.88:0/3823670549 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f2fa405f150 0x7f2fa4061610 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f2fa8009a80 tx=0x7f2fa801a000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:43.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.575+0000 7f2f9ffff700 1 -- 172.21.15.88:0/3823670549 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f2fac049ac0 con 0x7f2fb815c650 2023-10-18T16:07:43.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:43.781+0000 7f2fbea20700 1 -- 172.21.15.88:0/3823670549 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) v1 -- 0x7f2fb8147fd0 con 0x7f2fb815c650 2023-10-18T16:07:44.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.425+0000 7f2f9ffff700 1 -- 172.21.15.88:0/3823670549 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v6) v1 ==== 105+0+0 (secure 0 0 0) 0x7f2fac07be60 con 0x7f2fb815c650 2023-10-18T16:07:44.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.427+0000 7f2fbea20700 1 -- 172.21.15.88:0/3823670549 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f2fa405f150 msgr2=0x7f2fa4061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:44.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.427+0000 7f2fbea20700 1 --2- 172.21.15.88:0/3823670549 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f2fa405f150 0x7f2fa4061610 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f2fa8009a80 tx=0x7f2fa801a000).stop 2023-10-18T16:07:44.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.427+0000 7f2fbea20700 1 -- 172.21.15.88:0/3823670549 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2fb815c650 msgr2=0x7f2fb8147ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:44.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.427+0000 7f2fbea20700 1 --2- 172.21.15.88:0/3823670549 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2fb815c650 0x7f2fb8147ae0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f2fac001eb0 tx=0x7f2fac009040).stop 2023-10-18T16:07:44.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.427+0000 7f2fbea20700 1 -- 172.21.15.88:0/3823670549 shutdown_connections 2023-10-18T16:07:44.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.427+0000 7f2fbea20700 1 --2- 172.21.15.88:0/3823670549 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f2fa405f150 0x7f2fa4061610 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:44.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.427+0000 7f2fbea20700 1 --2- 172.21.15.88:0/3823670549 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2fb815f0e0 0x7f2fb80bc280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:44.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.427+0000 7f2fbea20700 1 --2- 172.21.15.88:0/3823670549 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2fa00038e0 0x7f2fb80c36c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-10-18T16:07:44.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.427+0000 7f2fbea20700 1 --2- 172.21.15.88:0/3823670549 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2fb815c650 0x7f2fb8147ae0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:44.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.427+0000 7f2fbea20700 1 -- 172.21.15.88:0/3823670549 >> 172.21.15.88:0/3823670549 conn(0x7f2fb8065cd0 msgr2=0x7f2fb806a410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:44.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.427+0000 7f2fbea20700 1 -- 172.21.15.88:0/3823670549 shutdown_connections 2023-10-18T16:07:44.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.427+0000 7f2fbea20700 1 -- 172.21.15.88:0/3823670549 wait complete. 2023-10-18T16:07:44.444 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-18T16:07:44.562 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.558+0000 7fcf5456f700 1 Processor -- start 2023-10-18T16:07:44.562 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.558+0000 7fcf5456f700 1 -- start start 2023-10-18T16:07:44.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.559+0000 7fcf5456f700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c14cfd0 0x7fcf4c0c3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:44.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.559+0000 7fcf5456f700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c06a670 0x7fcf4c150fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:44.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.559+0000 7fcf5456f700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf4c067560 con 0x7fcf4c0c38c0 2023-10-18T16:07:44.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.559+0000 7fcf5456f700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf4c1514a0 con 0x7fcf4c06a670 2023-10-18T16:07:44.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.559+0000 7fcf5456f700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf4c0be710 con 0x7fcf4c14cfd0 2023-10-18T16:07:44.566 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.559+0000 7fcf52b0c700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c06a670 0x7fcf4c150fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:44.566 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.559+0000 7fcf52b0c700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c06a670 0x7fcf4c150fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43988/0 (socket says 172.21.15.88:43988) 2023-10-18T16:07:44.566 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.559+0000 7fcf52b0c700 1 -- 172.21.15.88:0/2474841200 learned_addr learned my addr 172.21.15.88:0/2474841200 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:44.566 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.560+0000 7fcf51b0a700 1 --2- 172.21.15.88:0/2474841200 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c14cfd0 0x7fcf4c0c3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:44.566 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.560+0000 7fcf52b0c700 1 -- 172.21.15.88:0/2474841200 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c14cfd0 msgr2=0x7fcf4c0c3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:44.567 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.560+0000 7fcf52b0c700 1 --2- 172.21.15.88:0/2474841200 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c14cfd0 0x7fcf4c0c3ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:44.567 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.560+0000 7fcf52b0c700 1 -- 172.21.15.88:0/2474841200 >> v1:172.21.15.151:6789/0 conn(0x7fcf4c0c38c0 legacy=0x7fcf4c067cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:44.567 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.560+0000 7fcf52b0c700 1 -- 172.21.15.88:0/2474841200 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf4c067840 con 0x7fcf4c06a670 2023-10-18T16:07:44.567 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.560+0000 7fcf52b0c700 1 --2- 172.21.15.88:0/2474841200 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf4c06a670 0x7fcf4c150fb0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fcf44009f50 tx=0x7fcf44009840).ready entity=mon.0 client_cookie=352bd01bca2935ef server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:44.567 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.560+0000 7fcf51309700 1 -- 172.21.15.88:0/2474841200 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf4400e070 con 0x7fcf4c06a670 2023-10-18T16:07:44.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.560+0000 7fcf51309700 1 -- 172.21.15.88:0/2474841200 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf4400ba70 con 0x7fcf4c06a670 2023-10-18T16:07:44.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.560+0000 7fcf51309700 1 -- 172.21.15.88:0/2474841200 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf4400bdf0 con 0x7fcf4c06a670 2023-10-18T16:07:44.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.560+0000 7fcf5456f700 1 -- 172.21.15.88:0/2474841200 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf4c06a670 msgr2=0x7fcf4c150fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:44.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.560+0000 7fcf5456f700 1 --2- 172.21.15.88:0/2474841200 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf4c06a670 0x7fcf4c150fb0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fcf44009f50 tx=0x7fcf44009840).stop 2023-10-18T16:07:44.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.560+0000 7fcf5456f700 1 -- 172.21.15.88:0/2474841200 shutdown_connections 2023-10-18T16:07:44.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.560+0000 7fcf5456f700 1 --2- 172.21.15.88:0/2474841200 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf4c06a670 0x7fcf4c150fb0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:44.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.560+0000 7fcf5456f700 1 --2- 172.21.15.88:0/2474841200 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c14cfd0 0x7fcf4c0c3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:44.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.560+0000 7fcf5456f700 1 -- 172.21.15.88:0/2474841200 >> 172.21.15.88:0/2474841200 conn(0x7fcf4c065cd0 msgr2=0x7fcf4c0660e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:44.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.560+0000 7fcf5456f700 1 -- 172.21.15.88:0/2474841200 shutdown_connections 2023-10-18T16:07:44.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.561+0000 7fcf5456f700 1 -- 172.21.15.88:0/2474841200 wait complete. 2023-10-18T16:07:44.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.561+0000 7fcf5456f700 1 Processor -- start 2023-10-18T16:07:44.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.562+0000 7fcf5456f700 1 -- start start 2023-10-18T16:07:44.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.562+0000 7fcf5456f700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c06a670 0x7fcf4c149b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:44.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.562+0000 7fcf5456f700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf4c0c38c0 0x7fcf4c0c1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:44.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.562+0000 7fcf5456f700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcf4c14cfd0 0x7fcf4c0c1d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:44.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.562+0000 7fcf5456f700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf4c0c2220 con 0x7fcf4c0c38c0 2023-10-18T16:07:44.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.562+0000 7fcf5456f700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf4c14e470 con 0x7fcf4c14cfd0 2023-10-18T16:07:44.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.562+0000 7fcf5456f700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf4c14e6c0 con 0x7fcf4c06a670 2023-10-18T16:07:44.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.562+0000 7fcf51b0a700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf4c0c38c0 0x7fcf4c0c1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:44.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.562+0000 7fcf51b0a700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf4c0c38c0 0x7fcf4c0c1840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44004/0 (socket says 172.21.15.88:44004) 2023-10-18T16:07:44.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.562+0000 7fcf51b0a700 1 -- 172.21.15.88:0/2138644876 learned_addr learned my addr 172.21.15.88:0/2138644876 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:44.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.562+0000 7fcf5230b700 1 --2- 172.21.15.88:0/2138644876 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c06a670 0x7fcf4c149b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:44.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.562+0000 7fcf52b0c700 1 --2- 172.21.15.88:0/2138644876 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcf4c14cfd0 0x7fcf4c0c1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:44.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.563+0000 7fcf51b0a700 1 -- 172.21.15.88:0/2138644876 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c06a670 msgr2=0x7fcf4c149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:44.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.563+0000 7fcf51b0a700 1 --2- 172.21.15.88:0/2138644876 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c06a670 0x7fcf4c149b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:44.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.563+0000 7fcf51b0a700 1 -- 172.21.15.88:0/2138644876 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcf4c14cfd0 msgr2=0x7fcf4c0c1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:44.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.563+0000 7fcf51b0a700 1 --2- 172.21.15.88:0/2138644876 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcf4c14cfd0 0x7fcf4c0c1d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:44.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.563+0000 7fcf51b0a700 1 -- 172.21.15.88:0/2138644876 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf44009ae0 con 0x7fcf4c0c38c0 2023-10-18T16:07:44.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.563+0000 7fcf51b0a700 1 --2- 172.21.15.88:0/2138644876 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf4c0c38c0 0x7fcf4c0c1840 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fcf3c00d8c0 tx=0x7fcf3c008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:44.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.563+0000 7fcf427fc700 1 -- 172.21.15.88:0/2138644876 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf3c011590 con 0x7fcf4c0c38c0 2023-10-18T16:07:44.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.563+0000 7fcf427fc700 1 -- 172.21.15.88:0/2138644876 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf3c0116f0 con 0x7fcf4c0c38c0 2023-10-18T16:07:44.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.563+0000 7fcf5456f700 1 -- 172.21.15.88:0/2138644876 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf4c148a40 con 0x7fcf4c0c38c0 2023-10-18T16:07:44.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.563+0000 7fcf427fc700 1 -- 172.21.15.88:0/2138644876 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf3c011a70 con 0x7fcf4c0c38c0 2023-10-18T16:07:44.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.563+0000 7fcf5456f700 1 -- 172.21.15.88:0/2138644876 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf4c14fe70 con 0x7fcf4c0c38c0 2023-10-18T16:07:44.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.564+0000 7fcf5456f700 1 -- 172.21.15.88:0/2138644876 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf4c06c4e0 con 0x7fcf4c0c38c0 2023-10-18T16:07:44.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.564+0000 7fcf427fc700 1 -- 172.21.15.88:0/2138644876 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7fcf3c012420 con 0x7fcf4c0c38c0 2023-10-18T16:07:44.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.564+0000 7fcf427fc700 1 --2- 172.21.15.88:0/2138644876 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fcf3005f150 0x7fcf30061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:44.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.564+0000 7fcf427fc700 1 -- 172.21.15.88:0/2138644876 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6121+0+0 (secure 0 0 0) 0x7fcf3c079a80 con 0x7fcf4c0c38c0 2023-10-18T16:07:44.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.565+0000 7fcf5230b700 1 --2- 172.21.15.88:0/2138644876 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fcf3005f150 0x7fcf30061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:44.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.565+0000 7fcf5230b700 1 --2- 172.21.15.88:0/2138644876 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fcf3005f150 0x7fcf30061610 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fcf48003900 tx=0x7fcf48007d40).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:44.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.568+0000 7fcf427fc700 1 -- 172.21.15.88:0/2138644876 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7fcf3c04acf0 con 0x7fcf4c0c38c0 2023-10-18T16:07:44.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:44.774+0000 7fcf5456f700 1 -- 172.21.15.88:0/2138644876 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fcf4c06b140 con 0x7fcf4c0c38c0 2023-10-18T16:07:45.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.435+0000 7fcf427fc700 1 -- 172.21.15.88:0/2138644876 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v7) v1 ==== 122+0+0 (secure 0 0 0) 0x7fcf3c04e310 con 0x7fcf4c0c38c0 2023-10-18T16:07:45.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.437+0000 7fcf5456f700 1 -- 172.21.15.88:0/2138644876 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fcf3005f150 msgr2=0x7fcf30061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:45.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.437+0000 7fcf5456f700 1 --2- 172.21.15.88:0/2138644876 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fcf3005f150 0x7fcf30061610 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fcf48003900 tx=0x7fcf48007d40).stop 2023-10-18T16:07:45.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.437+0000 7fcf5456f700 1 -- 172.21.15.88:0/2138644876 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf4c0c38c0 msgr2=0x7fcf4c0c1840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:45.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.437+0000 7fcf5456f700 1 --2- 172.21.15.88:0/2138644876 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf4c0c38c0 0x7fcf4c0c1840 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fcf3c00d8c0 tx=0x7fcf3c008040).stop 2023-10-18T16:07:45.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.438+0000 7fcf5456f700 1 -- 172.21.15.88:0/2138644876 shutdown_connections 2023-10-18T16:07:45.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.438+0000 7fcf5456f700 1 --2- 172.21.15.88:0/2138644876 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fcf3005f150 0x7fcf30061610 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:45.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.438+0000 7fcf5456f700 1 --2- 172.21.15.88:0/2138644876 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcf4c14cfd0 0x7fcf4c0c1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:45.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.438+0000 7fcf5456f700 1 --2- 172.21.15.88:0/2138644876 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c06a670 0x7fcf4c149b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:45.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.438+0000 7fcf5456f700 1 --2- 172.21.15.88:0/2138644876 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf4c0c38c0 0x7fcf4c0c1840 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:45.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.438+0000 7fcf5456f700 1 -- 172.21.15.88:0/2138644876 >> 172.21.15.88:0/2138644876 conn(0x7fcf4c065cd0 msgr2=0x7fcf4c0bc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:45.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.438+0000 7fcf5456f700 1 -- 172.21.15.88:0/2138644876 shutdown_connections 2023-10-18T16:07:45.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.438+0000 7fcf5456f700 1 -- 172.21.15.88:0/2138644876 wait complete. 2023-10-18T16:07:45.455 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2023-10-18T16:07:45.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.573+0000 7f10fe0d5700 1 Processor -- start 2023-10-18T16:07:45.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.573+0000 7f10fe0d5700 1 -- start start 2023-10-18T16:07:45.579 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.574+0000 7f10fe0d5700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f10f80a6c70 0x7f10f8150ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:45.579 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.574+0000 7f10fe0d5700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f10f80a8580 0x7f10f80a8a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:45.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.574+0000 7f10fe0d5700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10f8151a10 con 0x7f10f80a6890 2023-10-18T16:07:45.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.574+0000 7f10fe0d5700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10f80a8f00 con 0x7f10f80a6c70 2023-10-18T16:07:45.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.574+0000 7f10fe0d5700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10f80a9310 con 0x7f10f80a8580 2023-10-18T16:07:45.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.574+0000 7f10f77fe700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f10f80a6890 0x7f10f8064340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:52392/0 (socket says 172.21.15.88:52392) 2023-10-18T16:07:45.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.574+0000 7f10f77fe700 1 -- 172.21.15.88:0/1155919605 learned_addr learned my addr 172.21.15.88:0/1155919605 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:07:45.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.574+0000 7f10f6ffd700 1 --2- 172.21.15.88:0/1155919605 >> v2:172.21.15.88:3300/0 conn(0x7f10f80a6c70 0x7f10f8150ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:45.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.574+0000 7f10f7fff700 1 --2- 172.21.15.88:0/1155919605 >> v2:172.21.15.151:3300/0 conn(0x7f10f80a8580 0x7f10f80a8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:45.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.574+0000 7f10f6ffd700 1 -- 172.21.15.88:0/1155919605 >> v2:172.21.15.151:3300/0 conn(0x7f10f80a8580 msgr2=0x7f10f80a8a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:45.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.574+0000 7f10f6ffd700 1 --2- 172.21.15.88:0/1155919605 >> v2:172.21.15.151:3300/0 conn(0x7f10f80a8580 0x7f10f80a8a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:45.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.574+0000 7f10f6ffd700 1 -- 172.21.15.88:0/1155919605 >> v1:172.21.15.88:6789/0 conn(0x7f10f80a6890 legacy=0x7f10f8064340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:45.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.574+0000 7f10f67fc700 1 -- 172.21.15.88:0/1155919605 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 582825853 0 0) 0x7f10f8151a10 con 0x7f10f80a6890 2023-10-18T16:07:45.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.574+0000 7f10f6ffd700 1 -- 172.21.15.88:0/1155919605 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10f81513c0 con 0x7f10f80a6c70 2023-10-18T16:07:45.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.575+0000 7f10f6ffd700 1 --2- 172.21.15.88:0/1155919605 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10f80a6c70 0x7f10f8150ed0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f10f805f670 tx=0x7f10ec009060).ready entity=mon.0 client_cookie=82f480cff899fdff server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:45.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.575+0000 7f10f67fc700 1 -- 172.21.15.88:0/1155919605 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10ec00b580 con 0x7f10f80a6c70 2023-10-18T16:07:45.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.575+0000 7f10f67fc700 1 -- 172.21.15.88:0/1155919605 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10ec00b6e0 con 0x7f10f80a6c70 2023-10-18T16:07:45.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.575+0000 7f10f67fc700 1 -- 172.21.15.88:0/1155919605 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10ec00ba60 con 0x7f10f80a6c70 2023-10-18T16:07:45.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.575+0000 7f10fe0d5700 1 -- 172.21.15.88:0/1155919605 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10f80a6c70 msgr2=0x7f10f8150ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:45.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.575+0000 7f10fe0d5700 1 --2- 172.21.15.88:0/1155919605 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10f80a6c70 0x7f10f8150ed0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f10f805f670 tx=0x7f10ec009060).stop 2023-10-18T16:07:45.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.575+0000 7f10fe0d5700 1 -- 172.21.15.88:0/1155919605 shutdown_connections 2023-10-18T16:07:45.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.575+0000 7f10fe0d5700 1 --2- 172.21.15.88:0/1155919605 >> v2:172.21.15.151:3300/0 conn(0x7f10f80a8580 0x7f10f80a8a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:45.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.575+0000 7f10fe0d5700 1 --2- 172.21.15.88:0/1155919605 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10f80a6c70 0x7f10f8150ed0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:45.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.575+0000 7f10fe0d5700 1 -- 172.21.15.88:0/1155919605 >> 172.21.15.88:0/1155919605 conn(0x7f10f8069d80 msgr2=0x7f10f806a190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:45.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.575+0000 7f10fe0d5700 1 -- 172.21.15.88:0/1155919605 shutdown_connections 2023-10-18T16:07:45.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.575+0000 7f10fe0d5700 1 -- 172.21.15.88:0/1155919605 wait complete. 2023-10-18T16:07:45.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.576+0000 7f10fe0d5700 1 Processor -- start 2023-10-18T16:07:45.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.577+0000 7f10fe0d5700 1 -- start start 2023-10-18T16:07:45.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.577+0000 7f10fe0d5700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f10f80a6890 0x7f10f80bbba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:45.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.577+0000 7f10fe0d5700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10f80a6c70 0x7f10f8152eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:45.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.577+0000 7f10fe0d5700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f10f80a8580 0x7f10f81533a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:45.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.577+0000 7f10fe0d5700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10f80abb50 con 0x7f10f80a6c70 2023-10-18T16:07:45.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.577+0000 7f10fe0d5700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10f80bb420 con 0x7f10f80a6890 2023-10-18T16:07:45.589 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.577+0000 7f10fe0d5700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10f80ab870 con 0x7f10f80a8580 2023-10-18T16:07:45.589 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.577+0000 7f10f6ffd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10f80a6c70 0x7f10f8152eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:45.589 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.577+0000 7f10f6ffd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10f80a6c70 0x7f10f8152eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44022/0 (socket says 172.21.15.88:44022) 2023-10-18T16:07:45.589 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.577+0000 7f10f6ffd700 1 -- 172.21.15.88:0/2697071964 learned_addr learned my addr 172.21.15.88:0/2697071964 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:45.589 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.578+0000 7f10f77fe700 1 --2- 172.21.15.88:0/2697071964 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f10f80a6890 0x7f10f80bbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:45.590 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.578+0000 7f10f6ffd700 1 -- 172.21.15.88:0/2697071964 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f10f80a8580 msgr2=0x7f10f81533a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:45.590 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.578+0000 7f10f6ffd700 1 --2- 172.21.15.88:0/2697071964 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f10f80a8580 0x7f10f81533a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-10-18T16:07:45.590 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.578+0000 7f10f6ffd700 1 -- 172.21.15.88:0/2697071964 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f10f80a6890 msgr2=0x7f10f80bbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:45.590 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.578+0000 7f10f6ffd700 1 --2- 172.21.15.88:0/2697071964 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f10f80a6890 0x7f10f80bbba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:45.590 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.578+0000 7f10f6ffd700 1 -- 172.21.15.88:0/2697071964 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10ec0092a0 con 0x7f10f80a6c70 2023-10-18T16:07:45.590 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.578+0000 7f10f6ffd700 1 --2- 172.21.15.88:0/2697071964 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10f80a6c70 0x7f10f8152eb0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f10f80843b0 tx=0x7f10ec017000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:45.591 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.578+0000 7f10df7fe700 1 -- 172.21.15.88:0/2697071964 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10ec00b850 con 0x7f10f80a6c70 2023-10-18T16:07:45.591 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.578+0000 7f10df7fe700 1 -- 172.21.15.88:0/2697071964 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10ec00b9b0 con 0x7f10f80a6c70 2023-10-18T16:07:45.591 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.578+0000 7f10fe0d5700 1 -- 172.21.15.88:0/2697071964 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10f80badd0 con 0x7f10f80a6c70 2023-10-18T16:07:45.591 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.578+0000 7f10df7fe700 1 -- 172.21.15.88:0/2697071964 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10ec01b690 con 0x7f10f80a6c70 2023-10-18T16:07:45.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.578+0000 7f10fe0d5700 1 -- 172.21.15.88:0/2697071964 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10f8149300 con 0x7f10f80a6c70 2023-10-18T16:07:45.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.579+0000 7f10fe0d5700 1 -- 172.21.15.88:0/2697071964 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10f80bac30 con 0x7f10f80a6c70 2023-10-18T16:07:45.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.579+0000 7f10df7fe700 1 -- 172.21.15.88:0/2697071964 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f10ec01bac0 con 0x7f10f80a6c70 2023-10-18T16:07:45.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.579+0000 7f10df7fe700 1 --2- 172.21.15.88:0/2697071964 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f10e405f150 0x7f10e4061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:45.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.579+0000 7f10df7fe700 1 -- 172.21.15.88:0/2697071964 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6121+0+0 (secure 0 0 0) 0x7f10ec07bad0 con 0x7f10f80a6c70 2023-10-18T16:07:45.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.580+0000 7f10f77fe700 1 --2- 172.21.15.88:0/2697071964 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f10e405f150 0x7f10e4061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:45.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.580+0000 7f10f77fe700 1 --2- 172.21.15.88:0/2697071964 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f10e405f150 0x7f10e4061610 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f10e8007fd0 tx=0x7f10e8009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:45.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.583+0000 7f10df7fe700 1 -- 172.21.15.88:0/2697071964 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f10f80bac30 con 0x7f10f80a6c70 2023-10-18T16:07:45.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:45.790+0000 7f10fe0d5700 1 -- 172.21.15.88:0/2697071964 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7f10f80bac30 con 0x7f10f80a6c70 2023-10-18T16:07:46.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.436+0000 7f10df7fe700 1 -- 172.21.15.88:0/2697071964 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v8) v1 ==== 115+0+0 (secure 0 0 0) 0x7f10f80bac30 con 0x7f10f80a6c70 2023-10-18T16:07:46.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.438+0000 7f10fe0d5700 1 -- 172.21.15.88:0/2697071964 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f10e405f150 msgr2=0x7f10e4061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:46.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.438+0000 7f10fe0d5700 1 --2- 172.21.15.88:0/2697071964 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f10e405f150 0x7f10e4061610 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f10e8007fd0 tx=0x7f10e8009040).stop 2023-10-18T16:07:46.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.438+0000 7f10fe0d5700 1 -- 172.21.15.88:0/2697071964 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10f80a6c70 msgr2=0x7f10f8152eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:46.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.438+0000 7f10fe0d5700 1 --2- 172.21.15.88:0/2697071964 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10f80a6c70 0x7f10f8152eb0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f10f80843b0 tx=0x7f10ec017000).stop 2023-10-18T16:07:46.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.439+0000 7f10fe0d5700 1 -- 172.21.15.88:0/2697071964 shutdown_connections 2023-10-18T16:07:46.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.439+0000 7f10fe0d5700 1 --2- 172.21.15.88:0/2697071964 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f10e405f150 0x7f10e4061610 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:46.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.439+0000 7f10fe0d5700 1 --2- 172.21.15.88:0/2697071964 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f10f80a8580 0x7f10f81533a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-10-18T16:07:46.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.439+0000 7f10fe0d5700 1 --2- 172.21.15.88:0/2697071964 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f10f80a6890 0x7f10f80bbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:46.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.439+0000 7f10fe0d5700 1 --2- 172.21.15.88:0/2697071964 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10f80a6c70 0x7f10f8152eb0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:46.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.439+0000 7f10fe0d5700 1 -- 172.21.15.88:0/2697071964 >> 172.21.15.88:0/2697071964 conn(0x7f10f8069d80 msgr2=0x7f10f8150c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:46.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.439+0000 7f10fe0d5700 1 -- 172.21.15.88:0/2697071964 shutdown_connections 2023-10-18T16:07:46.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.439+0000 7f10fe0d5700 1 -- 172.21.15.88:0/2697071964 wait complete. 2023-10-18T16:07:46.455 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-18T16:07:46.456 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-18T16:07:46.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.567+0000 7f74cc65f700 1 Processor -- start 2023-10-18T16:07:46.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.567+0000 7f74cc65f700 1 -- start start 2023-10-18T16:07:46.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.568+0000 7f74cc65f700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f74c40c20a0 0x7f74c4154280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:46.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.568+0000 7f74cc65f700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74c40a8570 con 0x7f74c414d880 2023-10-18T16:07:46.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.568+0000 7f74cc65f700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74c4154770 con 0x7f74c4150340 2023-10-18T16:07:46.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.568+0000 7f74cc65f700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74c40a87c0 con 0x7f74c40c20a0 2023-10-18T16:07:46.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.568+0000 7f74ca3fb700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f74c414d880 0x7f74c40c1990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:52402/0 (socket says 172.21.15.88:52402) 2023-10-18T16:07:46.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.568+0000 7f74ca3fb700 1 -- 172.21.15.88:0/1050264258 learned_addr learned my addr 172.21.15.88:0/1050264258 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:07:46.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.568+0000 7f74c9bfa700 1 --2- 172.21.15.88:0/1050264258 >> v2:172.21.15.151:3300/0 conn(0x7f74c40c20a0 0x7f74c4154280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:46.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.568+0000 7f74c93f9700 1 -- 172.21.15.88:0/1050264258 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2097097038 0 0) 0x7f74c40a8570 con 0x7f74c414d880 2023-10-18T16:07:46.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.568+0000 7f74c93f9700 1 -- 172.21.15.88:0/1050264258 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74b00033f0 con 0x7f74c414d880 2023-10-18T16:07:46.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.568+0000 7f74c93f9700 1 -- 172.21.15.88:0/1050264258 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3986017165 0 0) 0x7f74b00033f0 con 0x7f74c414d880 2023-10-18T16:07:46.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.568+0000 7f74c93f9700 1 -- 172.21.15.88:0/1050264258 >> v2:172.21.15.151:3300/0 conn(0x7f74c40c20a0 msgr2=0x7f74c4154280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:46.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.568+0000 7f74c93f9700 1 --2- 172.21.15.88:0/1050264258 >> v2:172.21.15.151:3300/0 conn(0x7f74c40c20a0 0x7f74c4154280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:46.579 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.568+0000 7f74c93f9700 1 -- 172.21.15.88:0/1050264258 >> v1:172.21.15.151:6789/0 conn(0x7f74c4150340 legacy=0x7f74c41507d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:46.579 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.568+0000 7f74c93f9700 1 -- 172.21.15.88:0/1050264258 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74c40a8aa0 con 0x7f74c414d880 2023-10-18T16:07:46.579 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.568+0000 7f74c93f9700 1 -- 172.21.15.88:0/1050264258 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441715075 0 0) 0x7f74bc0017f0 con 0x7f74c414d880 2023-10-18T16:07:46.579 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.568+0000 7f74c93f9700 1 -- 172.21.15.88:0/1050264258 >> v1:172.21.15.88:6789/0 conn(0x7f74c414d880 legacy=0x7f74c40c1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:46.579 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.568+0000 7f74c93f9700 1 --2- 172.21.15.88:0/1050264258 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f74b00038e0 0x7f74b0003cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:46.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.568+0000 7f74c93f9700 1 --2- 172.21.15.88:0/1050264258 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f74b0004690 0x7f74b0004b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:46.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.568+0000 7f74c93f9700 1 --2- 172.21.15.88:0/1050264258 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74b0005000 0x7f74b000b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:46.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.568+0000 7f74c93f9700 1 -- 172.21.15.88:0/1050264258 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c40a87c0 con 0x7f74b0005000 2023-10-18T16:07:46.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.568+0000 7f74c93f9700 1 -- 172.21.15.88:0/1050264258 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b000be00 con 0x7f74b00038e0 2023-10-18T16:07:46.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.568+0000 7f74c93f9700 1 -- 172.21.15.88:0/1050264258 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74b000bfc0 con 0x7f74b0004690 2023-10-18T16:07:46.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.568+0000 7f74cabfc700 1 --2- 172.21.15.88:0/1050264258 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74b0005000 0x7f74b000b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:46.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.569+0000 7f74ca3fb700 1 --2- 172.21.15.88:0/1050264258 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f74b00038e0 0x7f74b0003cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:46.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.569+0000 7f74c9bfa700 1 --2- 172.21.15.88:0/1050264258 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f74b0004690 0x7f74b0004b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:46.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.569+0000 7f74cabfc700 1 -- 172.21.15.88:0/1050264258 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f74b0004690 msgr2=0x7f74b0004b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:46.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.569+0000 7f74cabfc700 1 --2- 172.21.15.88:0/1050264258 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f74b0004690 0x7f74b0004b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:46.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.569+0000 7f74cabfc700 1 -- 172.21.15.88:0/1050264258 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f74b00038e0 msgr2=0x7f74b0003cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:46.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.569+0000 7f74cabfc700 1 --2- 172.21.15.88:0/1050264258 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f74b00038e0 0x7f74b0003cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:46.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.569+0000 7f74cabfc700 1 -- 172.21.15.88:0/1050264258 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74b00033f0 con 0x7f74b0005000 2023-10-18T16:07:46.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.569+0000 7f74cabfc700 1 -- 172.21.15.88:0/1050264258 reap_dead start 2023-10-18T16:07:46.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.569+0000 7f74cabfc700 1 --2- 172.21.15.88:0/1050264258 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74b0005000 0x7f74b000b8e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f74c0001820 tx=0x7f74c0007b90).ready entity=mon.0 client_cookie=dbf87ea3bea5f988 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:46.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.569+0000 7f74c93f9700 1 -- 172.21.15.88:0/1050264258 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c0007ea0 con 0x7f74b0005000 2023-10-18T16:07:46.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.569+0000 7f74c93f9700 1 -- 172.21.15.88:0/1050264258 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74c0012400 con 0x7f74b0005000 2023-10-18T16:07:46.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.569+0000 7f74c93f9700 1 -- 172.21.15.88:0/1050264258 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c0012780 con 0x7f74b0005000 2023-10-18T16:07:46.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.569+0000 7f74cc65f700 1 -- 172.21.15.88:0/1050264258 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74b0005000 msgr2=0x7f74b000b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:46.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.569+0000 7f74cc65f700 1 --2- 172.21.15.88:0/1050264258 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74b0005000 0x7f74b000b8e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f74c0001820 tx=0x7f74c0007b90).stop 2023-10-18T16:07:46.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.570+0000 7f74cc65f700 1 -- 172.21.15.88:0/1050264258 shutdown_connections 2023-10-18T16:07:46.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.570+0000 7f74cc65f700 1 --2- 172.21.15.88:0/1050264258 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74b0005000 0x7f74b000b8e0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:46.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.570+0000 7f74cc65f700 1 -- 172.21.15.88:0/1050264258 >> 172.21.15.88:0/1050264258 conn(0x7f74c4069d80 msgr2=0x7f74c406a190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:46.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.570+0000 7f74cc65f700 1 -- 172.21.15.88:0/1050264258 shutdown_connections 2023-10-18T16:07:46.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.570+0000 7f74cc65f700 1 -- 172.21.15.88:0/1050264258 wait complete. 2023-10-18T16:07:46.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.571+0000 7f74cc65f700 1 Processor -- start 2023-10-18T16:07:46.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.571+0000 7f74cc65f700 1 -- start start 2023-10-18T16:07:46.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.571+0000 7f74cc65f700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f74b0005000 0x7f74c414f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:46.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.571+0000 7f74cc65f700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f74c414f5c0 0x7f74c40bb8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:46.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.571+0000 7f74cc65f700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74c4153ba0 0x7f74c40bbdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:46.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.571+0000 7f74cc65f700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c4153150 con 0x7f74c4153ba0 2023-10-18T16:07:46.586 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.571+0000 7f74cc65f700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c414e770 con 0x7f74b0005000 2023-10-18T16:07:46.586 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.571+0000 7f74cc65f700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74c40bc2b0 con 0x7f74c414f5c0 2023-10-18T16:07:46.586 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.571+0000 7f74cabfc700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74c4153ba0 0x7f74c40bbdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:46.586 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.571+0000 7f74cabfc700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74c4153ba0 0x7f74c40bbdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44040/0 (socket says 172.21.15.88:44040) 2023-10-18T16:07:46.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.571+0000 7f74cabfc700 1 -- 172.21.15.88:0/596980906 learned_addr learned my addr 172.21.15.88:0/596980906 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:46.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.572+0000 7f74c9bfa700 1 --2- 172.21.15.88:0/596980906 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f74c414f5c0 0x7f74c40bb8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:46.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.572+0000 7f74ca3fb700 1 --2- 172.21.15.88:0/596980906 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f74b0005000 0x7f74c414f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:46.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.572+0000 7f74cabfc700 1 -- 172.21.15.88:0/596980906 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f74c414f5c0 msgr2=0x7f74c40bb8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:46.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.572+0000 7f74cabfc700 1 --2- 172.21.15.88:0/596980906 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f74c414f5c0 0x7f74c40bb8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:46.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.572+0000 7f74cabfc700 1 -- 172.21.15.88:0/596980906 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f74b0005000 msgr2=0x7f74c414f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:46.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.572+0000 7f74cabfc700 1 --2- 172.21.15.88:0/596980906 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f74b0005000 0x7f74c414f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:46.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.572+0000 7f74cabfc700 1 -- 172.21.15.88:0/596980906 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74c4160130 con 0x7f74c4153ba0 2023-10-18T16:07:46.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.572+0000 7f74ca3fb700 1 --2- 172.21.15.88:0/596980906 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f74b0005000 0x7f74c414f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:07:46.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.572+0000 7f74cabfc700 1 --2- 172.21.15.88:0/596980906 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74c4153ba0 0x7f74c40bbdc0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f74c0009000 tx=0x7f74c0016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:46.589 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.572+0000 7f74ba7fc700 1 -- 172.21.15.88:0/596980906 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c0007ea0 con 0x7f74c4153ba0 2023-10-18T16:07:46.589 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.572+0000 7f74ba7fc700 1 -- 172.21.15.88:0/596980906 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74c0011b10 con 0x7f74c4153ba0 2023-10-18T16:07:46.589 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.572+0000 7f74ba7fc700 1 -- 172.21.15.88:0/596980906 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c001d420 con 0x7f74c4153ba0 2023-10-18T16:07:46.589 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.572+0000 7f74cc65f700 1 -- 172.21.15.88:0/596980906 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74c4160410 con 0x7f74c4153ba0 2023-10-18T16:07:46.589 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.572+0000 7f74cc65f700 1 -- 172.21.15.88:0/596980906 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74c41609d0 con 0x7f74c4153ba0 2023-10-18T16:07:46.590 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.573+0000 7f74cc65f700 1 -- 172.21.15.88:0/596980906 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74c406c4e0 con 0x7f74c4153ba0 2023-10-18T16:07:46.591 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.573+0000 7f74ba7fc700 1 -- 172.21.15.88:0/596980906 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f74c001d810 con 0x7f74c4153ba0 2023-10-18T16:07:46.591 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.574+0000 7f74ba7fc700 1 --2- 172.21.15.88:0/596980906 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f74a805f120 0x7f74a80615e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:46.591 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.574+0000 7f74ba7fc700 1 -- 172.21.15.88:0/596980906 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6121+0+0 (secure 0 0 0) 0x7f74c007afb0 con 0x7f74c4153ba0 2023-10-18T16:07:46.591 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.574+0000 7f74ca3fb700 1 --2- 172.21.15.88:0/596980906 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f74a805f120 0x7f74a80615e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:46.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.574+0000 7f74ca3fb700 1 --2- 172.21.15.88:0/596980906 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f74a805f120 0x7f74a80615e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f74bc001580 tx=0x7f74bc009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:46.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.578+0000 7f74ba7fc700 1 -- 172.21.15.88:0/596980906 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f74c004c1c0 con 0x7f74c4153ba0 2023-10-18T16:07:46.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.781+0000 7f74cc65f700 1 -- 172.21.15.88:0/596980906 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f74c40a8d80 con 0x7f74c4153ba0 2023-10-18T16:07:46.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.782+0000 7f74ba7fc700 1 -- 172.21.15.88:0/596980906 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6004 (secure 0 0 0) 0x7f74c004f7e0 con 0x7f74c4153ba0 2023-10-18T16:07:46.786 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:07:46.786 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/3094207741","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3094207741},{"type":"v1","addr":"172.21.15.88:6833","nonce":3094207741}]},"join_fscid":-1,"export_targets":[],"features":4540138314316775423,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4272,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/1543332826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":1543332826},{"type":"v1","addr":"172.21.15.88:6835","nonce":1543332826}]},"join_fscid":-1,"export_targets":[],"features":4540138314316775423,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4279,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6836/941009236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6835","nonce":941009236},{"type":"v1","addr":"172.21.15.151:6836","nonce":941009236}]},"join_fscid":-1,"export_targets":[],"features":4540138314316775423,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/2987875245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2987875245},{"type":"v1","addr":"172.21.15.88:6837","nonce":2987875245}]},"join_fscid":-1,"export_targets":[],"features":4540138314316775423,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4288,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6839/2615482496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6838","nonce":2615482496},{"type":"v1","addr":"172.21.15.151:6839","nonce":2615482496}]},"join_fscid":-1,"export_targets":[],"features":4540138314316775423,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-18T16:07:41.782544+0000","modified":"2023-10-18T16:07:46.435067+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"d","rank":0,"incarnation":4,"state":"up:creating","state_seq":1,"addr":"172.21.15.151:6837/3689661027","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6834","nonce":3689661027},{"type":"v1","addr":"172.21.15.151:6837","nonce":3689661027}]},"join_fscid":-1,"export_targets":[],"features":4540138314316775423,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-18T16:07:46.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.783+0000 7f74cc65f700 1 -- 172.21.15.88:0/596980906 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f74a805f120 msgr2=0x7f74a80615e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:46.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.783+0000 7f74cc65f700 1 --2- 172.21.15.88:0/596980906 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f74a805f120 0x7f74a80615e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f74bc001580 tx=0x7f74bc009040).stop 2023-10-18T16:07:46.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.783+0000 7f74cc65f700 1 -- 172.21.15.88:0/596980906 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74c4153ba0 msgr2=0x7f74c40bbdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:46.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.783+0000 7f74cc65f700 1 --2- 172.21.15.88:0/596980906 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74c4153ba0 0x7f74c40bbdc0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f74c0009000 tx=0x7f74c0016000).stop 2023-10-18T16:07:46.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.783+0000 7f74cc65f700 1 -- 172.21.15.88:0/596980906 shutdown_connections 2023-10-18T16:07:46.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.783+0000 7f74cc65f700 1 --2- 172.21.15.88:0/596980906 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f74a805f120 0x7f74a80615e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:46.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.783+0000 7f74cc65f700 1 --2- 172.21.15.88:0/596980906 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74c4153ba0 0x7f74c40bbdc0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:46.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.783+0000 7f74cc65f700 1 --2- 172.21.15.88:0/596980906 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f74b0005000 0x7f74c414f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:46.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.783+0000 7f74cc65f700 1 --2- 172.21.15.88:0/596980906 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f74c414f5c0 0x7f74c40bb8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:46.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.783+0000 7f74cc65f700 1 -- 172.21.15.88:0/596980906 >> 172.21.15.88:0/596980906 conn(0x7f74c4069d80 msgr2=0x7f74c414d050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:46.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.783+0000 7f74cc65f700 1 -- 172.21.15.88:0/596980906 shutdown_connections 2023-10-18T16:07:46.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.783+0000 7f74cc65f700 1 -- 172.21.15.88:0/596980906 wait complete. 2023-10-18T16:07:46.790 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 8 2023-10-18T16:07:46.799 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-18T16:07:46.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.908+0000 7fd7ee46b700 1 Processor -- start 2023-10-18T16:07:46.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.908+0000 7fd7ee46b700 1 -- start start 2023-10-18T16:07:46.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.909+0000 7fd7ee46b700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd7e80c20a0 0x7fd7e8154280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:46.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.909+0000 7fd7ee46b700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd7e8150340 0x7fd7e81507f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:46.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.909+0000 7fd7ee46b700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7e80a8570 con 0x7fd7e814d880 2023-10-18T16:07:46.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.909+0000 7fd7ee46b700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7e8154770 con 0x7fd7e80c20a0 2023-10-18T16:07:46.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.909+0000 7fd7ee46b700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7e80a87c0 con 0x7fd7e8150340 2023-10-18T16:07:46.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.909+0000 7fd7e77fe700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd7e80c20a0 0x7fd7e8154280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:46.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.909+0000 7fd7e77fe700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd7e80c20a0 0x7fd7e8154280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44048/0 (socket says 172.21.15.88:44048) 2023-10-18T16:07:46.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.909+0000 7fd7e77fe700 1 -- 172.21.15.88:0/1520381378 learned_addr learned my addr 172.21.15.88:0/1520381378 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:46.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.909+0000 7fd7e7fff700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fd7e814d880 0x7fd7e80c1990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:52414/0 (socket says 172.21.15.88:52414) 2023-10-18T16:07:46.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.909+0000 7fd7eca08700 1 --2- 172.21.15.88:0/1520381378 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd7e8150340 0x7fd7e81507f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:46.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.910+0000 7fd7e77fe700 1 -- 172.21.15.88:0/1520381378 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd7e8150340 msgr2=0x7fd7e81507f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:46.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.910+0000 7fd7e77fe700 1 --2- 172.21.15.88:0/1520381378 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd7e8150340 0x7fd7e81507f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:46.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.910+0000 7fd7e77fe700 1 -- 172.21.15.88:0/1520381378 >> v1:172.21.15.88:6789/0 conn(0x7fd7e814d880 legacy=0x7fd7e80c1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:46.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.910+0000 7fd7e77fe700 1 -- 172.21.15.88:0/1520381378 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7e80a8aa0 con 0x7fd7e80c20a0 2023-10-18T16:07:46.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.910+0000 7fd7e6ffd700 1 -- 172.21.15.88:0/1520381378 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 480630760 0 0) 0x7fd7e80a8570 con 0x7fd7e814d880 2023-10-18T16:07:46.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.910+0000 7fd7eca08700 1 --2- 172.21.15.88:0/1520381378 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd7e8150340 0x7fd7e81507f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-10-18T16:07:46.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.910+0000 7fd7e77fe700 1 --2- 172.21.15.88:0/1520381378 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd7e80c20a0 0x7fd7e8154280 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fd7dc006fd0 tx=0x7fd7dc0090f0).ready entity=mon.0 client_cookie=2243bb49906b6631 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:46.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.910+0000 7fd7e6ffd700 1 -- 172.21.15.88:0/1520381378 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7dc00b600 con 0x7fd7e80c20a0 2023-10-18T16:07:46.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.910+0000 7fd7e6ffd700 1 -- 172.21.15.88:0/1520381378 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7dc00b760 con 0x7fd7e80c20a0 2023-10-18T16:07:46.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.910+0000 7fd7e6ffd700 1 -- 172.21.15.88:0/1520381378 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7dc00bae0 con 0x7fd7e80c20a0 2023-10-18T16:07:46.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.910+0000 7fd7ee46b700 1 -- 172.21.15.88:0/1520381378 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd7e80c20a0 msgr2=0x7fd7e8154280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:46.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.910+0000 7fd7ee46b700 1 --2- 172.21.15.88:0/1520381378 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd7e80c20a0 0x7fd7e8154280 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fd7dc006fd0 tx=0x7fd7dc0090f0).stop 2023-10-18T16:07:46.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.910+0000 7fd7ee46b700 1 -- 172.21.15.88:0/1520381378 shutdown_connections 2023-10-18T16:07:46.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.910+0000 7fd7ee46b700 1 --2- 172.21.15.88:0/1520381378 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd7e8150340 0x7fd7e81507f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:46.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.910+0000 7fd7ee46b700 1 --2- 172.21.15.88:0/1520381378 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd7e80c20a0 0x7fd7e8154280 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:46.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.910+0000 7fd7ee46b700 1 -- 172.21.15.88:0/1520381378 >> 172.21.15.88:0/1520381378 conn(0x7fd7e8069d80 msgr2=0x7fd7e806a190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:46.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.910+0000 7fd7ee46b700 1 -- 172.21.15.88:0/1520381378 shutdown_connections 2023-10-18T16:07:46.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.910+0000 7fd7ee46b700 1 -- 172.21.15.88:0/1520381378 wait complete. 2023-10-18T16:07:46.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.911+0000 7fd7ee46b700 1 Processor -- start 2023-10-18T16:07:46.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.912+0000 7fd7ee46b700 1 -- start start 2023-10-18T16:07:46.923 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.912+0000 7fd7ee46b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd7e80c20a0 0x7fd7e814f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:46.923 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.912+0000 7fd7ee46b700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd7e814d880 0x7fd7e8161860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:46.923 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.912+0000 7fd7ee46b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd7e8150340 0x7fd7e8161d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:46.924 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.912+0000 7fd7ee46b700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7e8162240 con 0x7fd7e8150340 2023-10-18T16:07:46.924 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.912+0000 7fd7ee46b700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7e80ac660 con 0x7fd7e80c20a0 2023-10-18T16:07:46.924 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.912+0000 7fd7ee46b700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7e8164640 con 0x7fd7e814d880 2023-10-18T16:07:46.924 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.912+0000 7fd7eca08700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd7e8150340 0x7fd7e8161d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:46.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.912+0000 7fd7eca08700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd7e8150340 0x7fd7e8161d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44052/0 (socket says 172.21.15.88:44052) 2023-10-18T16:07:46.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.912+0000 7fd7eca08700 1 -- 172.21.15.88:0/3160683525 learned_addr learned my addr 172.21.15.88:0/3160683525 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:46.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.913+0000 7fd7e77fe700 1 --2- 172.21.15.88:0/3160683525 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd7e814d880 0x7fd7e8161860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:46.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.913+0000 7fd7e7fff700 1 --2- 172.21.15.88:0/3160683525 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd7e80c20a0 0x7fd7e814f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:46.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.913+0000 7fd7eca08700 1 -- 172.21.15.88:0/3160683525 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd7e814d880 msgr2=0x7fd7e8161860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:46.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.913+0000 7fd7eca08700 1 --2- 172.21.15.88:0/3160683525 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd7e814d880 0x7fd7e8161860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:46.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.913+0000 7fd7eca08700 1 -- 172.21.15.88:0/3160683525 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd7e80c20a0 msgr2=0x7fd7e814f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:46.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.913+0000 7fd7eca08700 1 --2- 172.21.15.88:0/3160683525 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd7e80c20a0 0x7fd7e814f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:46.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.913+0000 7fd7eca08700 1 -- 172.21.15.88:0/3160683525 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7dc009370 con 0x7fd7e8150340 2023-10-18T16:07:46.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.913+0000 7fd7e77fe700 1 --2- 172.21.15.88:0/3160683525 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd7e814d880 0x7fd7e8161860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:07:46.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.913+0000 7fd7e7fff700 1 --2- 172.21.15.88:0/3160683525 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd7e80c20a0 0x7fd7e814f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:07:46.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.913+0000 7fd7eca08700 1 --2- 172.21.15.88:0/3160683525 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd7e8150340 0x7fd7e8161d50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7fd7e000cef0 tx=0x7fd7e0007860).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:46.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.913+0000 7fd7cffff700 1 -- 172.21.15.88:0/3160683525 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7e000ae00 con 0x7fd7e8150340 2023-10-18T16:07:46.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.913+0000 7fd7cffff700 1 -- 172.21.15.88:0/3160683525 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7e0016400 con 0x7fd7e8150340 2023-10-18T16:07:46.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.913+0000 7fd7ee46b700 1 -- 172.21.15.88:0/3160683525 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7e8160420 con 0x7fd7e8150340 2023-10-18T16:07:46.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.913+0000 7fd7cffff700 1 -- 172.21.15.88:0/3160683525 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7e0016780 con 0x7fd7e8150340 2023-10-18T16:07:46.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.913+0000 7fd7ee46b700 1 -- 172.21.15.88:0/3160683525 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7e81609e0 con 0x7fd7e8150340 2023-10-18T16:07:46.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.914+0000 7fd7ee46b700 1 -- 172.21.15.88:0/3160683525 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7e806c4e0 con 0x7fd7e8150340 2023-10-18T16:07:46.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.914+0000 7fd7cffff700 1 -- 172.21.15.88:0/3160683525 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7fd7e001f3f0 con 0x7fd7e8150340 2023-10-18T16:07:46.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.914+0000 7fd7cffff700 1 --2- 172.21.15.88:0/3160683525 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fd7d405f150 0x7fd7d4061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:46.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.915+0000 7fd7cffff700 1 -- 172.21.15.88:0/3160683525 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6121+0+0 (secure 0 0 0) 0x7fd7e007a940 con 0x7fd7e8150340 2023-10-18T16:07:46.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.915+0000 7fd7e7fff700 1 --2- 172.21.15.88:0/3160683525 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fd7d405f150 0x7fd7d4061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:46.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.915+0000 7fd7e7fff700 1 --2- 172.21.15.88:0/3160683525 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fd7d405f150 0x7fd7d4061610 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fd7d80011e0 tx=0x7fd7d8007b10).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:46.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:46.918+0000 7fd7cffff700 1 -- 172.21.15.88:0/3160683525 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7fd7e004bbb0 con 0x7fd7e8150340 2023-10-18T16:07:47.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.081+0000 7fd7ee46b700 1 -- 172.21.15.88:0/3160683525 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd7e80c2c00 con 0x7fd7e8150340 2023-10-18T16:07:47.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.082+0000 7fd7cffff700 1 -- 172.21.15.88:0/3160683525 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16928 (secure 0 0 0) 0x7fd7e001b070 con 0x7fd7e8150340 2023-10-18T16:07:47.086 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:07:47.086 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":22,"fsid":"fb1f77e8-ead0-4f90-8128-fdf5137eb16f","created":"2023-10-18T16:07:23.662468+0000","modified":"2023-10-18T16:07:42.784120+0000","last_up_change":"2023-10-18T16:07:32.498203+0000","last_in_change":"2023-10-18T16:07:26.064784+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-10-18T16:07:32.536320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-18T16:07:34.769989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-10-18T16:07:39.439702+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-10-18T16:07:40.753774+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e421c20c-f7bc-4574-97a2-201bb4a16b4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":91007},{"type":"v1","addr":"172.21.15.88:6809","nonce":91007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":91007},{"type":"v1","addr":"172.21.15.88:6811","nonce":91007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":91007},{"type":"v1","addr":"172.21.15.88:6815","nonce":91007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":91007},{"type":"v1","addr":"172.21.15.88:6813","nonce":91007}]},"public_addr":"172.21.15.88:6809/91007","cluster_addr":"172.21.15.88:6811/91007","heartbeat_back_addr":"172.21.15.88:6815/91007","heartbeat_front_addr":"172.21.15.88:6813/91007","state":["exists","up"]},{"osd":1,"uuid":"9485c5ec-379a-4038-a2d8-1f1081fe2eea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":91009},{"type":"v1","addr":"172.21.15.88:6825","nonce":91009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":91009},{"type":"v1","addr":"172.21.15.88:6827","nonce":91009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":91009},{"type":"v1","addr":"172.21.15.88:6831","nonce":91009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":91009},{"type":"v1","addr":"172.21.15.88:6829","nonce":91009}]},"public_addr":"172.21.15.88:6825/91009","cluster_addr":"172.21.15.88:6827/91009","heartbeat_back_addr":"172.21.15.88:6831/91009","heartbeat_front_addr":"172.21.15.88:6829/91009","state":["exists","up"]},{"osd":2,"uuid":"671895e2-57f3-4f1a-ba4a-a580af1f3ec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":91005},{"type":"v1","addr":"172.21.15.88:6801","nonce":91005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":91005},{"type":"v1","addr":"172.21.15.88:6803","nonce":91005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":91005},{"type":"v1","addr":"172.21.15.88:6807","nonce":91005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":91005},{"type":"v1","addr":"172.21.15.88:6805","nonce":91005}]},"public_addr":"172.21.15.88:6801/91005","cluster_addr":"172.21.15.88:6803/91005","heartbeat_back_addr":"172.21.15.88:6807/91005","heartbeat_front_addr":"172.21.15.88:6805/91005","state":["exists","up"]},{"osd":3,"uuid":"0c4e6e7f-7d9c-4e65-beec-fd0ca384f320","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":91008},{"type":"v1","addr":"172.21.15.88:6817","nonce":91008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":91008},{"type":"v1","addr":"172.21.15.88:6819","nonce":91008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":91008},{"type":"v1","addr":"172.21.15.88:6823","nonce":91008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":91008},{"type":"v1","addr":"172.21.15.88:6821","nonce":91008}]},"public_addr":"172.21.15.88:6817/91008","cluster_addr":"172.21.15.88:6819/91008","heartbeat_back_addr":"172.21.15.88:6823/91008","heartbeat_front_addr":"172.21.15.88:6821/91008","state":["exists","up"]},{"osd":4,"uuid":"29570691-ef35-4b80-9247-aa3703877af4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":91514},{"type":"v1","addr":"172.21.15.151:6801","nonce":91514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":91514},{"type":"v1","addr":"172.21.15.151:6803","nonce":91514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":91514},{"type":"v1","addr":"172.21.15.151:6807","nonce":91514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":91514},{"type":"v1","addr":"172.21.15.151:6805","nonce":91514}]},"public_addr":"172.21.15.151:6801/91514","cluster_addr":"172.21.15.151:6803/91514","heartbeat_back_addr":"172.21.15.151:6807/91514","heartbeat_front_addr":"172.21.15.151:6805/91514","state":["exists","up"]},{"osd":5,"uuid":"26540b7b-032a-4b3b-bf6c-b131a449fa99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":91513},{"type":"v1","addr":"172.21.15.151:6809","nonce":91513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":91513},{"type":"v1","addr":"172.21.15.151:6811","nonce":91513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":91513},{"type":"v1","addr":"172.21.15.151:6815","nonce":91513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":91513},{"type":"v1","addr":"172.21.15.151:6813","nonce":91513}]},"public_addr":"172.21.15.151:6809/91513","cluster_addr":"172.21.15.151:6811/91513","heartbeat_back_addr":"172.21.15.151:6815/91513","heartbeat_front_addr":"172.21.15.151:6813/91513","state":["exists","up"]},{"osd":6,"uuid":"7af01f9c-90db-45fd-93c8-09517c9b0881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":91544},{"type":"v1","addr":"172.21.15.151:6817","nonce":91544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":91544},{"type":"v1","addr":"172.21.15.151:6819","nonce":91544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":91544},{"type":"v1","addr":"172.21.15.151:6823","nonce":91544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":91544},{"type":"v1","addr":"172.21.15.151:6821","nonce":91544}]},"public_addr":"172.21.15.151:6817/91544","cluster_addr":"172.21.15.151:6819/91544","heartbeat_back_addr":"172.21.15.151:6823/91544","heartbeat_front_addr":"172.21.15.151:6821/91544","state":["exists","up"]},{"osd":7,"uuid":"b527b0a5-4249-4854-a30f-bff897757239","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":91545},{"type":"v1","addr":"172.21.15.151:6825","nonce":91545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":91545},{"type":"v1","addr":"172.21.15.151:6827","nonce":91545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":91545},{"type":"v1","addr":"172.21.15.151:6831","nonce":91545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":91545},{"type":"v1","addr":"172.21.15.151:6829","nonce":91545}]},"public_addr":"172.21.15.151:6825/91545","cluster_addr":"172.21.15.151:6827/91545","heartbeat_back_addr":"172.21.15.151:6831/91545","heartbeat_front_addr":"172.21.15.151:6829/91545","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T16:07:31.216742+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T16:07:31.467229+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T16:07:31.220809+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T16:07:31.414999+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T16:07:31.188976+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T16:07:31.252153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T16:07:31.254324+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T16:07:31.203434+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-18T16:07:47.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.084+0000 7fd7ee46b700 1 -- 172.21.15.88:0/3160683525 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fd7d405f150 msgr2=0x7fd7d4061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:47.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.084+0000 7fd7ee46b700 1 --2- 172.21.15.88:0/3160683525 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fd7d405f150 0x7fd7d4061610 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fd7d80011e0 tx=0x7fd7d8007b10).stop 2023-10-18T16:07:47.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.084+0000 7fd7ee46b700 1 -- 172.21.15.88:0/3160683525 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd7e8150340 msgr2=0x7fd7e8161d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:47.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.084+0000 7fd7ee46b700 1 --2- 172.21.15.88:0/3160683525 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd7e8150340 0x7fd7e8161d50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7fd7e000cef0 tx=0x7fd7e0007860).stop 2023-10-18T16:07:47.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.084+0000 7fd7ee46b700 1 -- 172.21.15.88:0/3160683525 shutdown_connections 2023-10-18T16:07:47.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.084+0000 7fd7ee46b700 1 --2- 172.21.15.88:0/3160683525 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fd7d405f150 0x7fd7d4061610 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:47.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.084+0000 7fd7ee46b700 1 --2- 172.21.15.88:0/3160683525 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd7e8150340 0x7fd7e8161d50 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:47.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.084+0000 7fd7ee46b700 1 --2- 172.21.15.88:0/3160683525 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd7e80c20a0 0x7fd7e814f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:47.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.084+0000 7fd7ee46b700 1 --2- 172.21.15.88:0/3160683525 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd7e814d880 0x7fd7e8161860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:47.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.084+0000 7fd7ee46b700 1 -- 172.21.15.88:0/3160683525 >> 172.21.15.88:0/3160683525 conn(0x7fd7e8069d80 msgr2=0x7fd7e814cf40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:47.090 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.084+0000 7fd7ee46b700 1 -- 172.21.15.88:0/3160683525 shutdown_connections 2023-10-18T16:07:47.090 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.084+0000 7fd7ee46b700 1 -- 172.21.15.88:0/3160683525 wait complete. 2023-10-18T16:07:47.100 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-18T16:07:47.100 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-18T16:07:47.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.209+0000 7ff63f370700 1 Processor -- start 2023-10-18T16:07:47.213 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.209+0000 7ff63f370700 1 -- start start 2023-10-18T16:07:47.215 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.210+0000 7ff63f370700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff638149d50 0x7ff638067110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:47.215 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.210+0000 7ff63f370700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7ff63814a130 0x7ff638067600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:47.215 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.210+0000 7ff63f370700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6381514b0 con 0x7ff638151940 2023-10-18T16:07:47.215 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.210+0000 7ff63f370700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6380badb0 con 0x7ff638149d50 2023-10-18T16:07:47.216 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.210+0000 7ff63f370700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff638068440 con 0x7ff63814a130 2023-10-18T16:07:47.216 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.210+0000 7ff63e36e700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff638149d50 0x7ff638067110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:47.216 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.210+0000 7ff63e36e700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff638149d50 0x7ff638067110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44068/0 (socket says 172.21.15.88:44068) 2023-10-18T16:07:47.216 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.210+0000 7ff63e36e700 1 -- 172.21.15.88:0/1166514434 learned_addr learned my addr 172.21.15.88:0/1166514434 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:47.216 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.210+0000 7ff63db6d700 1 --2- 172.21.15.88:0/1166514434 >> v2:172.21.15.151:3300/0 conn(0x7ff63814a130 0x7ff638067600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:47.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.210+0000 7ff63e36e700 1 -- 172.21.15.88:0/1166514434 >> v2:172.21.15.151:3300/0 conn(0x7ff63814a130 msgr2=0x7ff638067600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:47.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.210+0000 7ff63e36e700 1 --2- 172.21.15.88:0/1166514434 >> v2:172.21.15.151:3300/0 conn(0x7ff63814a130 0x7ff638067600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:47.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.210+0000 7ff63e36e700 1 -- 172.21.15.88:0/1166514434 >> v1:172.21.15.151:6789/0 conn(0x7ff638151940 legacy=0x7ff638151d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:47.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.210+0000 7ff63e36e700 1 -- 172.21.15.88:0/1166514434 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6380686e0 con 0x7ff638149d50 2023-10-18T16:07:47.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.210+0000 7ff63e36e700 1 --2- 172.21.15.88:0/1166514434 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff638149d50 0x7ff638067110 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7ff6380bb730 tx=0x7ff630009840).ready entity=mon.0 client_cookie=96fb3cd8b4c50824 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:47.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.210+0000 7ff63d36c700 1 -- 172.21.15.88:0/1166514434 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff63000e070 con 0x7ff638149d50 2023-10-18T16:07:47.218 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.210+0000 7ff63d36c700 1 -- 172.21.15.88:0/1166514434 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff63000ba20 con 0x7ff638149d50 2023-10-18T16:07:47.218 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.210+0000 7ff63d36c700 1 -- 172.21.15.88:0/1166514434 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff63000bda0 con 0x7ff638149d50 2023-10-18T16:07:47.218 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.211+0000 7ff63f370700 1 -- 172.21.15.88:0/1166514434 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff638149d50 msgr2=0x7ff638067110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:47.218 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.211+0000 7ff63f370700 1 --2- 172.21.15.88:0/1166514434 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff638149d50 0x7ff638067110 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7ff6380bb730 tx=0x7ff630009840).stop 2023-10-18T16:07:47.218 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.211+0000 7ff63f370700 1 -- 172.21.15.88:0/1166514434 shutdown_connections 2023-10-18T16:07:47.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.211+0000 7ff63f370700 1 --2- 172.21.15.88:0/1166514434 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff638149d50 0x7ff638067110 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:47.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.211+0000 7ff63f370700 1 --2- 172.21.15.88:0/1166514434 >> v2:172.21.15.151:3300/0 conn(0x7ff63814a130 0x7ff638067600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:47.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.211+0000 7ff63f370700 1 -- 172.21.15.88:0/1166514434 >> 172.21.15.88:0/1166514434 conn(0x7ff638062650 msgr2=0x7ff638062a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:47.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.211+0000 7ff63f370700 1 -- 172.21.15.88:0/1166514434 shutdown_connections 2023-10-18T16:07:47.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.211+0000 7ff63f370700 1 -- 172.21.15.88:0/1166514434 wait complete. 2023-10-18T16:07:47.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.212+0000 7ff63f370700 1 Processor -- start 2023-10-18T16:07:47.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.212+0000 7ff63f370700 1 -- start start 2023-10-18T16:07:47.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.213+0000 7ff63f370700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff638149d50 0x7ff63814ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:47.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.213+0000 7ff63f370700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff63814a130 0x7ff638147ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:47.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.213+0000 7ff63f370700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff638151940 0x7ff63814ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:47.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.213+0000 7ff63f370700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff63814dd50 con 0x7ff63814a130 2023-10-18T16:07:47.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.213+0000 7ff63f370700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff63814e8d0 con 0x7ff638151940 2023-10-18T16:07:47.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.213+0000 7ff63f370700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff63814fe60 con 0x7ff638149d50 2023-10-18T16:07:47.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.213+0000 7ff63db6d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff63814a130 0x7ff638147ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:47.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.213+0000 7ff63db6d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff63814a130 0x7ff638147ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44082/0 (socket says 172.21.15.88:44082) 2023-10-18T16:07:47.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.213+0000 7ff63db6d700 1 -- 172.21.15.88:0/3180231849 learned_addr learned my addr 172.21.15.88:0/3180231849 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:47.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.213+0000 7ff63e36e700 1 --2- 172.21.15.88:0/3180231849 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff638149d50 0x7ff63814ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:47.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.213+0000 7ff63eb6f700 1 --2- 172.21.15.88:0/3180231849 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff638151940 0x7ff63814ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:47.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.213+0000 7ff63db6d700 1 -- 172.21.15.88:0/3180231849 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff638149d50 msgr2=0x7ff63814ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:47.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.213+0000 7ff63db6d700 1 --2- 172.21.15.88:0/3180231849 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff638149d50 0x7ff63814ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:47.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.213+0000 7ff63db6d700 1 -- 172.21.15.88:0/3180231849 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff638151940 msgr2=0x7ff63814ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:47.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.213+0000 7ff63db6d700 1 --2- 172.21.15.88:0/3180231849 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff638151940 0x7ff63814ae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:47.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.213+0000 7ff63db6d700 1 -- 172.21.15.88:0/3180231849 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff630009ae0 con 0x7ff63814a130 2023-10-18T16:07:47.226 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.213+0000 7ff63db6d700 1 --2- 172.21.15.88:0/3180231849 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff63814a130 0x7ff638147ff0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7ff62c00e930 tx=0x7ff62c009060).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:47.226 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.213+0000 7ff6367fc700 1 -- 172.21.15.88:0/3180231849 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff62c012600 con 0x7ff63814a130 2023-10-18T16:07:47.226 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.213+0000 7ff6367fc700 1 -- 172.21.15.88:0/3180231849 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff62c012760 con 0x7ff63814a130 2023-10-18T16:07:47.226 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.213+0000 7ff63f370700 1 -- 172.21.15.88:0/3180231849 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff63814dfa0 con 0x7ff63814a130 2023-10-18T16:07:47.226 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.213+0000 7ff6367fc700 1 -- 172.21.15.88:0/3180231849 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff62c012ae0 con 0x7ff63814a130 2023-10-18T16:07:47.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.213+0000 7ff63f370700 1 -- 172.21.15.88:0/3180231849 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff63814e560 con 0x7ff63814a130 2023-10-18T16:07:47.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.214+0000 7ff63f370700 1 -- 172.21.15.88:0/3180231849 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff638069fd0 con 0x7ff63814a130 2023-10-18T16:07:47.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.215+0000 7ff6367fc700 1 -- 172.21.15.88:0/3180231849 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7ff62c01b400 con 0x7ff63814a130 2023-10-18T16:07:47.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.215+0000 7ff6367fc700 1 --2- 172.21.15.88:0/3180231849 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7ff62405f150 0x7ff624061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:47.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.215+0000 7ff6367fc700 1 -- 172.21.15.88:0/3180231849 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6121+0+0 (secure 0 0 0) 0x7ff62c07a5d0 con 0x7ff63814a130 2023-10-18T16:07:47.228 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.215+0000 7ff63e36e700 1 --2- 172.21.15.88:0/3180231849 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7ff62405f150 0x7ff624061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:47.228 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.216+0000 7ff63e36e700 1 --2- 172.21.15.88:0/3180231849 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7ff62405f150 0x7ff624061610 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7ff630009880 tx=0x7ff6380bb580).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:47.228 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.219+0000 7ff6367fc700 1 -- 172.21.15.88:0/3180231849 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7ff62c04b840 con 0x7ff63814a130 2023-10-18T16:07:47.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.375+0000 7ff63f370700 1 -- 172.21.15.88:0/3180231849 --> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff6380689c0 con 0x7ff62405f150 2023-10-18T16:07:47.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.386+0000 7ff6367fc700 1 -- 172.21.15.88:0/3180231849 <== mgr.4102 v2:172.21.15.151:6832/90916 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+336380 (secure 0 0 0) 0x7ff6380689c0 con 0x7ff62405f150 2023-10-18T16:07:47.393 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:07:47.396 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.389+0000 7ff63f370700 1 -- 172.21.15.88:0/3180231849 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7ff62405f150 msgr2=0x7ff624061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:47.396 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.389+0000 7ff63f370700 1 --2- 172.21.15.88:0/3180231849 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7ff62405f150 0x7ff624061610 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7ff630009880 tx=0x7ff6380bb580).stop 2023-10-18T16:07:47.397 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.389+0000 7ff63f370700 1 -- 172.21.15.88:0/3180231849 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff63814a130 msgr2=0x7ff638147ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:47.397 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.389+0000 7ff63f370700 1 --2- 172.21.15.88:0/3180231849 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff63814a130 0x7ff638147ff0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7ff62c00e930 tx=0x7ff62c009060).stop 2023-10-18T16:07:47.397 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.389+0000 7ff63f370700 1 -- 172.21.15.88:0/3180231849 shutdown_connections 2023-10-18T16:07:47.397 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.389+0000 7ff63f370700 1 --2- 172.21.15.88:0/3180231849 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7ff62405f150 0x7ff624061610 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:47.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.389+0000 7ff63f370700 1 --2- 172.21.15.88:0/3180231849 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff638151940 0x7ff63814ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:47.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.389+0000 7ff63f370700 1 --2- 172.21.15.88:0/3180231849 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff638149d50 0x7ff63814ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:47.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.389+0000 7ff63f370700 1 --2- 172.21.15.88:0/3180231849 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff63814a130 0x7ff638147ff0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:47.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.389+0000 7ff63f370700 1 -- 172.21.15.88:0/3180231849 >> 172.21.15.88:0/3180231849 conn(0x7ff638062650 msgr2=0x7ff638064760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:47.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.389+0000 7ff63f370700 1 -- 172.21.15.88:0/3180231849 shutdown_connections 2023-10-18T16:07:47.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.389+0000 7ff63f370700 1 -- 172.21.15.88:0/3180231849 wait complete. 2023-10-18T16:07:47.400 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2023-10-18T16:07:47.404 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2023-10-18T16:07:46.457808+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":274,"acting":274,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":98,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2378500,"kb_used_data":1732,"kb_used_omap":0,"kb_used_meta":2376704,"kb_avail":752596220,"statfs":{"total":773094113280,"available":770658529280,"internally_reserved":0,"allocated":1773568,"data_stored":649208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2433744896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"4.1d","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.800856+0000","last_change":"2023-10-18T16:07:41.796671+0000","last_active":"2023-10-18T16:07:41.800856+0000","last_peered":"2023-10-18T16:07:41.800856+0000","last_clean":"2023-10-18T16:07:41.800856+0000","last_became_active":"2023-10-18T16:07:41.796451+0000","last_became_peered":"2023-10-18T16:07:41.796451+0000","last_unstale":"2023-10-18T16:07:41.800856+0000","last_undegraded":"2023-10-18T16:07:41.800856+0000","last_fullsized":"2023-10-18T16:07:41.800856+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.794828+0000","last_change":"2023-10-18T16:07:41.442111+0000","last_active":"2023-10-18T16:07:41.794828+0000","last_peered":"2023-10-18T16:07:41.794828+0000","last_clean":"2023-10-18T16:07:41.794828+0000","last_became_active":"2023-10-18T16:07:41.441815+0000","last_became_peered":"2023-10-18T16:07:41.441815+0000","last_unstale":"2023-10-18T16:07:41.794828+0000","last_undegraded":"2023-10-18T16:07:41.794828+0000","last_fullsized":"2023-10-18T16:07:41.794828+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":8,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.808517+0000","last_change":"2023-10-18T16:07:41.808517+0000","last_active":"2023-10-18T16:07:41.808517+0000","last_peered":"2023-10-18T16:07:41.808517+0000","last_clean":"2023-10-18T16:07:41.808517+0000","last_became_active":"2023-10-18T16:07:41.808224+0000","last_became_peered":"2023-10-18T16:07:41.808224+0000","last_unstale":"2023-10-18T16:07:41.808517+0000","last_undegraded":"2023-10-18T16:07:41.808517+0000","last_fullsized":"2023-10-18T16:07:41.808517+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.433589+0000","last_change":"2023-10-18T16:07:41.430021+0000","last_active":"2023-10-18T16:07:41.433589+0000","last_peered":"2023-10-18T16:07:41.433589+0000","last_clean":"2023-10-18T16:07:41.433589+0000","last_became_active":"2023-10-18T16:07:41.429781+0000","last_became_peered":"2023-10-18T16:07:41.429781+0000","last_unstale":"2023-10-18T16:07:41.433589+0000","last_undegraded":"2023-10-18T16:07:41.433589+0000","last_fullsized":"2023-10-18T16:07:41.433589+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":8,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.807746+0000","last_change":"2023-10-18T16:07:41.807746+0000","last_active":"2023-10-18T16:07:41.807746+0000","last_peered":"2023-10-18T16:07:41.807746+0000","last_clean":"2023-10-18T16:07:41.807746+0000","last_became_active":"2023-10-18T16:07:41.807256+0000","last_became_peered":"2023-10-18T16:07:41.807256+0000","last_unstale":"2023-10-18T16:07:41.807746+0000","last_undegraded":"2023-10-18T16:07:41.807746+0000","last_fullsized":"2023-10-18T16:07:41.807746+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.788944+0000","last_change":"2023-10-18T16:07:41.431122+0000","last_active":"2023-10-18T16:07:41.788944+0000","last_peered":"2023-10-18T16:07:41.788944+0000","last_clean":"2023-10-18T16:07:41.788944+0000","last_became_active":"2023-10-18T16:07:41.430831+0000","last_became_peered":"2023-10-18T16:07:41.430831+0000","last_unstale":"2023-10-18T16:07:41.788944+0000","last_undegraded":"2023-10-18T16:07:41.788944+0000","last_fullsized":"2023-10-18T16:07:41.788944+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.807269+0000","last_change":"2023-10-18T16:07:41.801866+0000","last_active":"2023-10-18T16:07:41.807269+0000","last_peered":"2023-10-18T16:07:41.807269+0000","last_clean":"2023-10-18T16:07:41.807269+0000","last_became_active":"2023-10-18T16:07:41.801602+0000","last_became_peered":"2023-10-18T16:07:41.801602+0000","last_unstale":"2023-10-18T16:07:41.807269+0000","last_undegraded":"2023-10-18T16:07:41.807269+0000","last_fullsized":"2023-10-18T16:07:41.807269+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.443089+0000","last_change":"2023-10-18T16:07:41.442448+0000","last_active":"2023-10-18T16:07:41.443089+0000","last_peered":"2023-10-18T16:07:41.443089+0000","last_clean":"2023-10-18T16:07:41.443089+0000","last_became_active":"2023-10-18T16:07:41.442143+0000","last_became_peered":"2023-10-18T16:07:41.442143+0000","last_unstale":"2023-10-18T16:07:41.443089+0000","last_undegraded":"2023-10-18T16:07:41.443089+0000","last_fullsized":"2023-10-18T16:07:41.443089+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.808841+0000","last_change":"2023-10-18T16:07:41.808302+0000","last_active":"2023-10-18T16:07:41.808841+0000","last_peered":"2023-10-18T16:07:41.808841+0000","last_clean":"2023-10-18T16:07:41.808841+0000","last_became_active":"2023-10-18T16:07:41.808001+0000","last_became_peered":"2023-10-18T16:07:41.808001+0000","last_unstale":"2023-10-18T16:07:41.808841+0000","last_undegraded":"2023-10-18T16:07:41.808841+0000","last_fullsized":"2023-10-18T16:07:41.808841+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.441462+0000","last_change":"2023-10-18T16:07:41.433080+0000","last_active":"2023-10-18T16:07:41.441462+0000","last_peered":"2023-10-18T16:07:41.441462+0000","last_clean":"2023-10-18T16:07:41.441462+0000","last_became_active":"2023-10-18T16:07:41.432714+0000","last_became_peered":"2023-10-18T16:07:41.432714+0000","last_unstale":"2023-10-18T16:07:41.441462+0000","last_undegraded":"2023-10-18T16:07:41.441462+0000","last_fullsized":"2023-10-18T16:07:41.441462+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":8,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.797599+0000","last_change":"2023-10-18T16:07:41.797599+0000","last_active":"2023-10-18T16:07:41.797599+0000","last_peered":"2023-10-18T16:07:41.797599+0000","last_clean":"2023-10-18T16:07:41.797599+0000","last_became_active":"2023-10-18T16:07:41.797318+0000","last_became_peered":"2023-10-18T16:07:41.797318+0000","last_unstale":"2023-10-18T16:07:41.797599+0000","last_undegraded":"2023-10-18T16:07:41.797599+0000","last_fullsized":"2023-10-18T16:07:41.797599+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.798325+0000","last_change":"2023-10-18T16:07:41.437425+0000","last_active":"2023-10-18T16:07:41.798325+0000","last_peered":"2023-10-18T16:07:41.798325+0000","last_clean":"2023-10-18T16:07:41.798325+0000","last_became_active":"2023-10-18T16:07:41.437135+0000","last_became_peered":"2023-10-18T16:07:41.437135+0000","last_unstale":"2023-10-18T16:07:41.798325+0000","last_undegraded":"2023-10-18T16:07:41.798325+0000","last_fullsized":"2023-10-18T16:07:41.798325+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.806231+0000","last_change":"2023-10-18T16:07:41.802908+0000","last_active":"2023-10-18T16:07:41.806231+0000","last_peered":"2023-10-18T16:07:41.806231+0000","last_clean":"2023-10-18T16:07:41.806231+0000","last_became_active":"2023-10-18T16:07:41.802644+0000","last_became_peered":"2023-10-18T16:07:41.802644+0000","last_unstale":"2023-10-18T16:07:41.806231+0000","last_undegraded":"2023-10-18T16:07:41.806231+0000","last_fullsized":"2023-10-18T16:07:41.806231+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.796534+0000","last_change":"2023-10-18T16:07:41.446842+0000","last_active":"2023-10-18T16:07:41.796534+0000","last_peered":"2023-10-18T16:07:41.796534+0000","last_clean":"2023-10-18T16:07:41.796534+0000","last_became_active":"2023-10-18T16:07:41.446567+0000","last_became_peered":"2023-10-18T16:07:41.446567+0000","last_unstale":"2023-10-18T16:07:41.796534+0000","last_undegraded":"2023-10-18T16:07:41.796534+0000","last_fullsized":"2023-10-18T16:07:41.796534+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.802559+0000","last_change":"2023-10-18T16:07:41.798736+0000","last_active":"2023-10-18T16:07:41.802559+0000","last_peered":"2023-10-18T16:07:41.802559+0000","last_clean":"2023-10-18T16:07:41.802559+0000","last_became_active":"2023-10-18T16:07:41.798458+0000","last_became_peered":"2023-10-18T16:07:41.798458+0000","last_unstale":"2023-10-18T16:07:41.802559+0000","last_undegraded":"2023-10-18T16:07:41.802559+0000","last_fullsized":"2023-10-18T16:07:41.802559+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.796971+0000","last_change":"2023-10-18T16:07:41.443369+0000","last_active":"2023-10-18T16:07:41.796971+0000","last_peered":"2023-10-18T16:07:41.796971+0000","last_clean":"2023-10-18T16:07:41.796971+0000","last_became_active":"2023-10-18T16:07:41.443064+0000","last_became_peered":"2023-10-18T16:07:41.443064+0000","last_unstale":"2023-10-18T16:07:41.796971+0000","last_undegraded":"2023-10-18T16:07:41.796971+0000","last_fullsized":"2023-10-18T16:07:41.796971+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":8,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.808196+0000","last_change":"2023-10-18T16:07:41.808196+0000","last_active":"2023-10-18T16:07:41.808196+0000","last_peered":"2023-10-18T16:07:41.808196+0000","last_clean":"2023-10-18T16:07:41.808196+0000","last_became_active":"2023-10-18T16:07:41.807537+0000","last_became_peered":"2023-10-18T16:07:41.807537+0000","last_unstale":"2023-10-18T16:07:41.808196+0000","last_undegraded":"2023-10-18T16:07:41.808196+0000","last_fullsized":"2023-10-18T16:07:41.808196+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.435794+0000","last_change":"2023-10-18T16:07:41.433498+0000","last_active":"2023-10-18T16:07:41.435794+0000","last_peered":"2023-10-18T16:07:41.435794+0000","last_clean":"2023-10-18T16:07:41.435794+0000","last_became_active":"2023-10-18T16:07:41.433321+0000","last_became_peered":"2023-10-18T16:07:41.433321+0000","last_unstale":"2023-10-18T16:07:41.435794+0000","last_undegraded":"2023-10-18T16:07:41.435794+0000","last_fullsized":"2023-10-18T16:07:41.435794+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":8,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.797372+0000","last_change":"2023-10-18T16:07:41.797372+0000","last_active":"2023-10-18T16:07:41.797372+0000","last_peered":"2023-10-18T16:07:41.797372+0000","last_clean":"2023-10-18T16:07:41.797372+0000","last_became_active":"2023-10-18T16:07:41.797108+0000","last_became_peered":"2023-10-18T16:07:41.797108+0000","last_unstale":"2023-10-18T16:07:41.797372+0000","last_undegraded":"2023-10-18T16:07:41.797372+0000","last_fullsized":"2023-10-18T16:07:41.797372+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.799678+0000","last_change":"2023-10-18T16:07:41.445469+0000","last_active":"2023-10-18T16:07:41.799678+0000","last_peered":"2023-10-18T16:07:41.799678+0000","last_clean":"2023-10-18T16:07:41.799678+0000","last_became_active":"2023-10-18T16:07:41.445204+0000","last_became_peered":"2023-10-18T16:07:41.445204+0000","last_unstale":"2023-10-18T16:07:41.799678+0000","last_undegraded":"2023-10-18T16:07:41.799678+0000","last_fullsized":"2023-10-18T16:07:41.799678+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.808614+0000","last_change":"2023-10-18T16:07:41.808046+0000","last_active":"2023-10-18T16:07:41.808614+0000","last_peered":"2023-10-18T16:07:41.808614+0000","last_clean":"2023-10-18T16:07:41.808614+0000","last_became_active":"2023-10-18T16:07:41.807189+0000","last_became_peered":"2023-10-18T16:07:41.807189+0000","last_unstale":"2023-10-18T16:07:41.808614+0000","last_undegraded":"2023-10-18T16:07:41.808614+0000","last_fullsized":"2023-10-18T16:07:41.808614+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.444649+0000","last_change":"2023-10-18T16:07:41.434313+0000","last_active":"2023-10-18T16:07:41.444649+0000","last_peered":"2023-10-18T16:07:41.444649+0000","last_clean":"2023-10-18T16:07:41.444649+0000","last_became_active":"2023-10-18T16:07:41.434022+0000","last_became_peered":"2023-10-18T16:07:41.434022+0000","last_unstale":"2023-10-18T16:07:41.444649+0000","last_undegraded":"2023-10-18T16:07:41.444649+0000","last_fullsized":"2023-10-18T16:07:41.444649+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":8,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.800593+0000","last_change":"2023-10-18T16:07:41.800593+0000","last_active":"2023-10-18T16:07:41.800593+0000","last_peered":"2023-10-18T16:07:41.800593+0000","last_clean":"2023-10-18T16:07:41.800593+0000","last_became_active":"2023-10-18T16:07:41.800302+0000","last_became_peered":"2023-10-18T16:07:41.800302+0000","last_unstale":"2023-10-18T16:07:41.800593+0000","last_undegraded":"2023-10-18T16:07:41.800593+0000","last_fullsized":"2023-10-18T16:07:41.800593+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.798209+0000","last_change":"2023-10-18T16:07:41.442977+0000","last_active":"2023-10-18T16:07:41.798209+0000","last_peered":"2023-10-18T16:07:41.798209+0000","last_clean":"2023-10-18T16:07:41.798209+0000","last_became_active":"2023-10-18T16:07:41.442383+0000","last_became_peered":"2023-10-18T16:07:41.442383+0000","last_unstale":"2023-10-18T16:07:41.798209+0000","last_undegraded":"2023-10-18T16:07:41.798209+0000","last_fullsized":"2023-10-18T16:07:41.798209+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.810814+0000","last_change":"2023-10-18T16:07:41.808758+0000","last_active":"2023-10-18T16:07:41.810814+0000","last_peered":"2023-10-18T16:07:41.810814+0000","last_clean":"2023-10-18T16:07:41.810814+0000","last_became_active":"2023-10-18T16:07:41.808475+0000","last_became_peered":"2023-10-18T16:07:41.808475+0000","last_unstale":"2023-10-18T16:07:41.810814+0000","last_undegraded":"2023-10-18T16:07:41.810814+0000","last_fullsized":"2023-10-18T16:07:41.810814+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.798959+0000","last_change":"2023-10-18T16:07:41.442566+0000","last_active":"2023-10-18T16:07:41.798959+0000","last_peered":"2023-10-18T16:07:41.798959+0000","last_clean":"2023-10-18T16:07:41.798959+0000","last_became_active":"2023-10-18T16:07:41.442264+0000","last_became_peered":"2023-10-18T16:07:41.442264+0000","last_unstale":"2023-10-18T16:07:41.798959+0000","last_undegraded":"2023-10-18T16:07:41.798959+0000","last_fullsized":"2023-10-18T16:07:41.798959+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":8,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.803408+0000","last_change":"2023-10-18T16:07:41.803408+0000","last_active":"2023-10-18T16:07:41.803408+0000","last_peered":"2023-10-18T16:07:41.803408+0000","last_clean":"2023-10-18T16:07:41.803408+0000","last_became_active":"2023-10-18T16:07:41.803142+0000","last_became_peered":"2023-10-18T16:07:41.803142+0000","last_unstale":"2023-10-18T16:07:41.803408+0000","last_undegraded":"2023-10-18T16:07:41.803408+0000","last_fullsized":"2023-10-18T16:07:41.803408+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.799082+0000","last_change":"2023-10-18T16:07:41.440460+0000","last_active":"2023-10-18T16:07:41.799082+0000","last_peered":"2023-10-18T16:07:41.799082+0000","last_clean":"2023-10-18T16:07:41.799082+0000","last_became_active":"2023-10-18T16:07:41.440162+0000","last_became_peered":"2023-10-18T16:07:41.440162+0000","last_unstale":"2023-10-18T16:07:41.799082+0000","last_undegraded":"2023-10-18T16:07:41.799082+0000","last_fullsized":"2023-10-18T16:07:41.799082+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":8,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.804730+0000","last_change":"2023-10-18T16:07:41.804730+0000","last_active":"2023-10-18T16:07:41.804730+0000","last_peered":"2023-10-18T16:07:41.804730+0000","last_clean":"2023-10-18T16:07:41.804730+0000","last_became_active":"2023-10-18T16:07:41.804455+0000","last_became_peered":"2023-10-18T16:07:41.804455+0000","last_unstale":"2023-10-18T16:07:41.804730+0000","last_undegraded":"2023-10-18T16:07:41.804730+0000","last_fullsized":"2023-10-18T16:07:41.804730+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.435773+0000","last_change":"2023-10-18T16:07:41.429156+0000","last_active":"2023-10-18T16:07:41.435773+0000","last_peered":"2023-10-18T16:07:41.435773+0000","last_clean":"2023-10-18T16:07:41.435773+0000","last_became_active":"2023-10-18T16:07:41.428828+0000","last_became_peered":"2023-10-18T16:07:41.428828+0000","last_unstale":"2023-10-18T16:07:41.435773+0000","last_undegraded":"2023-10-18T16:07:41.435773+0000","last_fullsized":"2023-10-18T16:07:41.435773+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.811080+0000","last_change":"2023-10-18T16:07:41.810261+0000","last_active":"2023-10-18T16:07:41.811080+0000","last_peered":"2023-10-18T16:07:41.811080+0000","last_clean":"2023-10-18T16:07:41.811080+0000","last_became_active":"2023-10-18T16:07:41.809938+0000","last_became_peered":"2023-10-18T16:07:41.809938+0000","last_unstale":"2023-10-18T16:07:41.811080+0000","last_undegraded":"2023-10-18T16:07:41.811080+0000","last_fullsized":"2023-10-18T16:07:41.811080+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.788799+0000","last_change":"2023-10-18T16:07:41.436527+0000","last_active":"2023-10-18T16:07:41.788799+0000","last_peered":"2023-10-18T16:07:41.788799+0000","last_clean":"2023-10-18T16:07:41.788799+0000","last_became_active":"2023-10-18T16:07:41.436223+0000","last_became_peered":"2023-10-18T16:07:41.436223+0000","last_unstale":"2023-10-18T16:07:41.788799+0000","last_undegraded":"2023-10-18T16:07:41.788799+0000","last_fullsized":"2023-10-18T16:07:41.788799+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.806186+0000","last_change":"2023-10-18T16:07:41.802220+0000","last_active":"2023-10-18T16:07:41.806186+0000","last_peered":"2023-10-18T16:07:41.806186+0000","last_clean":"2023-10-18T16:07:41.806186+0000","last_became_active":"2023-10-18T16:07:41.801909+0000","last_became_peered":"2023-10-18T16:07:41.801909+0000","last_unstale":"2023-10-18T16:07:41.806186+0000","last_undegraded":"2023-10-18T16:07:41.806186+0000","last_fullsized":"2023-10-18T16:07:41.806186+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.447846+0000","last_change":"2023-10-18T16:07:41.446075+0000","last_active":"2023-10-18T16:07:41.447846+0000","last_peered":"2023-10-18T16:07:41.447846+0000","last_clean":"2023-10-18T16:07:41.447846+0000","last_became_active":"2023-10-18T16:07:41.445817+0000","last_became_peered":"2023-10-18T16:07:41.445817+0000","last_unstale":"2023-10-18T16:07:41.447846+0000","last_undegraded":"2023-10-18T16:07:41.447846+0000","last_fullsized":"2023-10-18T16:07:41.447846+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.806073+0000","last_change":"2023-10-18T16:07:41.802289+0000","last_active":"2023-10-18T16:07:41.806073+0000","last_peered":"2023-10-18T16:07:41.806073+0000","last_clean":"2023-10-18T16:07:41.806073+0000","last_became_active":"2023-10-18T16:07:41.802038+0000","last_became_peered":"2023-10-18T16:07:41.802038+0000","last_unstale":"2023-10-18T16:07:41.806073+0000","last_undegraded":"2023-10-18T16:07:41.806073+0000","last_fullsized":"2023-10-18T16:07:41.806073+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.795284+0000","last_change":"2023-10-18T16:07:41.445650+0000","last_active":"2023-10-18T16:07:41.795284+0000","last_peered":"2023-10-18T16:07:41.795284+0000","last_clean":"2023-10-18T16:07:41.795284+0000","last_became_active":"2023-10-18T16:07:41.445297+0000","last_became_peered":"2023-10-18T16:07:41.445297+0000","last_unstale":"2023-10-18T16:07:41.795284+0000","last_undegraded":"2023-10-18T16:07:41.795284+0000","last_fullsized":"2023-10-18T16:07:41.795284+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":8,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.800350+0000","last_change":"2023-10-18T16:07:41.800350+0000","last_active":"2023-10-18T16:07:41.800350+0000","last_peered":"2023-10-18T16:07:41.800350+0000","last_clean":"2023-10-18T16:07:41.800350+0000","last_became_active":"2023-10-18T16:07:41.800095+0000","last_became_peered":"2023-10-18T16:07:41.800095+0000","last_unstale":"2023-10-18T16:07:41.800350+0000","last_undegraded":"2023-10-18T16:07:41.800350+0000","last_fullsized":"2023-10-18T16:07:41.800350+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.789807+0000","last_change":"2023-10-18T16:07:41.429381+0000","last_active":"2023-10-18T16:07:41.789807+0000","last_peered":"2023-10-18T16:07:41.789807+0000","last_clean":"2023-10-18T16:07:41.789807+0000","last_became_active":"2023-10-18T16:07:41.429117+0000","last_became_peered":"2023-10-18T16:07:41.429117+0000","last_unstale":"2023-10-18T16:07:41.789807+0000","last_undegraded":"2023-10-18T16:07:41.789807+0000","last_fullsized":"2023-10-18T16:07:41.789807+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.417130+0000","last_change":"2023-10-18T16:07:37.254194+0000","last_active":"2023-10-18T16:07:41.417130+0000","last_peered":"2023-10-18T16:07:41.417130+0000","last_clean":"2023-10-18T16:07:41.417130+0000","last_became_active":"2023-10-18T16:07:37.253900+0000","last_became_peered":"2023-10-18T16:07:37.253900+0000","last_unstale":"2023-10-18T16:07:41.417130+0000","last_undegraded":"2023-10-18T16:07:41.417130+0000","last_fullsized":"2023-10-18T16:07:41.417130+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:35.522314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:35.522314+0000","last_clean_scrub_stamp":"2023-10-18T16:07:35.522314+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.441430+0000","last_change":"2023-10-18T16:07:41.436498+0000","last_active":"2023-10-18T16:07:41.441430+0000","last_peered":"2023-10-18T16:07:41.441430+0000","last_clean":"2023-10-18T16:07:41.441430+0000","last_became_active":"2023-10-18T16:07:41.436182+0000","last_became_peered":"2023-10-18T16:07:41.436182+0000","last_unstale":"2023-10-18T16:07:41.441430+0000","last_undegraded":"2023-10-18T16:07:41.441430+0000","last_fullsized":"2023-10-18T16:07:41.441430+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":8,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.793823+0000","last_change":"2023-10-18T16:07:41.793823+0000","last_active":"2023-10-18T16:07:41.793823+0000","last_peered":"2023-10-18T16:07:41.793823+0000","last_clean":"2023-10-18T16:07:41.793823+0000","last_became_active":"2023-10-18T16:07:41.792600+0000","last_became_peered":"2023-10-18T16:07:41.792600+0000","last_unstale":"2023-10-18T16:07:41.793823+0000","last_undegraded":"2023-10-18T16:07:41.793823+0000","last_fullsized":"2023-10-18T16:07:41.793823+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.424416+0000","last_change":"2023-10-18T16:07:36.544503+0000","last_active":"2023-10-18T16:07:41.424416+0000","last_peered":"2023-10-18T16:07:41.424416+0000","last_clean":"2023-10-18T16:07:41.424416+0000","last_became_active":"2023-10-18T16:07:36.544254+0000","last_became_peered":"2023-10-18T16:07:36.544254+0000","last_unstale":"2023-10-18T16:07:41.424416+0000","last_undegraded":"2023-10-18T16:07:41.424416+0000","last_fullsized":"2023-10-18T16:07:41.424416+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:35.522314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:35.522314+0000","last_clean_scrub_stamp":"2023-10-18T16:07:35.522314+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.795162+0000","last_change":"2023-10-18T16:07:41.437532+0000","last_active":"2023-10-18T16:07:41.795162+0000","last_peered":"2023-10-18T16:07:41.795162+0000","last_clean":"2023-10-18T16:07:41.795162+0000","last_became_active":"2023-10-18T16:07:41.437257+0000","last_became_peered":"2023-10-18T16:07:41.437257+0000","last_unstale":"2023-10-18T16:07:41.795162+0000","last_undegraded":"2023-10-18T16:07:41.795162+0000","last_fullsized":"2023-10-18T16:07:41.795162+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":8,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.801719+0000","last_change":"2023-10-18T16:07:41.801719+0000","last_active":"2023-10-18T16:07:41.801719+0000","last_peered":"2023-10-18T16:07:41.801719+0000","last_clean":"2023-10-18T16:07:41.801719+0000","last_became_active":"2023-10-18T16:07:41.801325+0000","last_became_peered":"2023-10-18T16:07:41.801325+0000","last_unstale":"2023-10-18T16:07:41.801719+0000","last_undegraded":"2023-10-18T16:07:41.801719+0000","last_fullsized":"2023-10-18T16:07:41.801719+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":8,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.791251+0000","last_change":"2023-10-18T16:07:41.791251+0000","last_active":"2023-10-18T16:07:41.791251+0000","last_peered":"2023-10-18T16:07:41.791251+0000","last_clean":"2023-10-18T16:07:41.791251+0000","last_became_active":"2023-10-18T16:07:41.790992+0000","last_became_peered":"2023-10-18T16:07:41.790992+0000","last_unstale":"2023-10-18T16:07:41.791251+0000","last_undegraded":"2023-10-18T16:07:41.791251+0000","last_fullsized":"2023-10-18T16:07:41.791251+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.795059+0000","last_change":"2023-10-18T16:07:36.542401+0000","last_active":"2023-10-18T16:07:41.795059+0000","last_peered":"2023-10-18T16:07:41.795059+0000","last_clean":"2023-10-18T16:07:41.795059+0000","last_became_active":"2023-10-18T16:07:36.542071+0000","last_became_peered":"2023-10-18T16:07:36.542071+0000","last_unstale":"2023-10-18T16:07:41.795059+0000","last_undegraded":"2023-10-18T16:07:41.795059+0000","last_fullsized":"2023-10-18T16:07:41.795059+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:35.522314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:35.522314+0000","last_clean_scrub_stamp":"2023-10-18T16:07:35.522314+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.433392+0000","last_change":"2023-10-18T16:07:41.428730+0000","last_active":"2023-10-18T16:07:41.433392+0000","last_peered":"2023-10-18T16:07:41.433392+0000","last_clean":"2023-10-18T16:07:41.433392+0000","last_became_active":"2023-10-18T16:07:41.428469+0000","last_became_peered":"2023-10-18T16:07:41.428469+0000","last_unstale":"2023-10-18T16:07:41.433392+0000","last_undegraded":"2023-10-18T16:07:41.433392+0000","last_fullsized":"2023-10-18T16:07:41.433392+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":8,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.798241+0000","last_change":"2023-10-18T16:07:41.798241+0000","last_active":"2023-10-18T16:07:41.798241+0000","last_peered":"2023-10-18T16:07:41.798241+0000","last_clean":"2023-10-18T16:07:41.798241+0000","last_became_active":"2023-10-18T16:07:41.797958+0000","last_became_peered":"2023-10-18T16:07:41.797958+0000","last_unstale":"2023-10-18T16:07:41.798241+0000","last_undegraded":"2023-10-18T16:07:41.798241+0000","last_fullsized":"2023-10-18T16:07:41.798241+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.422240+0000","last_change":"2023-10-18T16:07:36.542651+0000","last_active":"2023-10-18T16:07:41.422240+0000","last_peered":"2023-10-18T16:07:41.422240+0000","last_clean":"2023-10-18T16:07:41.422240+0000","last_became_active":"2023-10-18T16:07:36.542306+0000","last_became_peered":"2023-10-18T16:07:36.542306+0000","last_unstale":"2023-10-18T16:07:41.422240+0000","last_undegraded":"2023-10-18T16:07:41.422240+0000","last_fullsized":"2023-10-18T16:07:41.422240+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:35.522314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:35.522314+0000","last_clean_scrub_stamp":"2023-10-18T16:07:35.522314+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.795394+0000","last_change":"2023-10-18T16:07:34.567589+0000","last_active":"2023-10-18T16:07:41.795394+0000","last_peered":"2023-10-18T16:07:41.795394+0000","last_clean":"2023-10-18T16:07:41.795394+0000","last_became_active":"2023-10-18T16:07:34.567301+0000","last_became_peered":"2023-10-18T16:07:34.567301+0000","last_unstale":"2023-10-18T16:07:41.795394+0000","last_undegraded":"2023-10-18T16:07:41.795394+0000","last_fullsized":"2023-10-18T16:07:41.795394+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:33.506699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:33.506699+0000","last_clean_scrub_stamp":"2023-10-18T16:07:33.506699+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.788528+0000","last_change":"2023-10-18T16:07:41.438974+0000","last_active":"2023-10-18T16:07:41.788528+0000","last_peered":"2023-10-18T16:07:41.788528+0000","last_clean":"2023-10-18T16:07:41.788528+0000","last_became_active":"2023-10-18T16:07:41.438710+0000","last_became_peered":"2023-10-18T16:07:41.438710+0000","last_unstale":"2023-10-18T16:07:41.788528+0000","last_undegraded":"2023-10-18T16:07:41.788528+0000","last_fullsized":"2023-10-18T16:07:41.788528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":8,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.800880+0000","last_change":"2023-10-18T16:07:41.800880+0000","last_active":"2023-10-18T16:07:41.800880+0000","last_peered":"2023-10-18T16:07:41.800880+0000","last_clean":"2023-10-18T16:07:41.800880+0000","last_became_active":"2023-10-18T16:07:41.800478+0000","last_became_peered":"2023-10-18T16:07:41.800478+0000","last_unstale":"2023-10-18T16:07:41.800880+0000","last_undegraded":"2023-10-18T16:07:41.800880+0000","last_fullsized":"2023-10-18T16:07:41.800880+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.795831+0000","last_change":"2023-10-18T16:07:36.542454+0000","last_active":"2023-10-18T16:07:41.795831+0000","last_peered":"2023-10-18T16:07:41.795831+0000","last_clean":"2023-10-18T16:07:41.795831+0000","last_became_active":"2023-10-18T16:07:36.542134+0000","last_became_peered":"2023-10-18T16:07:36.542134+0000","last_unstale":"2023-10-18T16:07:41.795831+0000","last_undegraded":"2023-10-18T16:07:41.795831+0000","last_fullsized":"2023-10-18T16:07:41.795831+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:35.522314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:35.522314+0000","last_clean_scrub_stamp":"2023-10-18T16:07:35.522314+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.446922+0000","last_change":"2023-10-18T16:07:41.445653+0000","last_active":"2023-10-18T16:07:41.446922+0000","last_peered":"2023-10-18T16:07:41.446922+0000","last_clean":"2023-10-18T16:07:41.446922+0000","last_became_active":"2023-10-18T16:07:41.445399+0000","last_became_peered":"2023-10-18T16:07:41.445399+0000","last_unstale":"2023-10-18T16:07:41.446922+0000","last_undegraded":"2023-10-18T16:07:41.446922+0000","last_fullsized":"2023-10-18T16:07:41.446922+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.798965+0000","last_change":"2023-10-18T16:07:37.254490+0000","last_active":"2023-10-18T16:07:41.798965+0000","last_peered":"2023-10-18T16:07:41.798965+0000","last_clean":"2023-10-18T16:07:41.798965+0000","last_became_active":"2023-10-18T16:07:37.254182+0000","last_became_peered":"2023-10-18T16:07:37.254182+0000","last_unstale":"2023-10-18T16:07:41.798965+0000","last_undegraded":"2023-10-18T16:07:41.798965+0000","last_fullsized":"2023-10-18T16:07:41.798965+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:35.522314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:35.522314+0000","last_clean_scrub_stamp":"2023-10-18T16:07:35.522314+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.433424+0000","last_change":"2023-10-18T16:07:41.428721+0000","last_active":"2023-10-18T16:07:41.433424+0000","last_peered":"2023-10-18T16:07:41.433424+0000","last_clean":"2023-10-18T16:07:41.433424+0000","last_became_active":"2023-10-18T16:07:41.428460+0000","last_became_peered":"2023-10-18T16:07:41.428460+0000","last_unstale":"2023-10-18T16:07:41.433424+0000","last_undegraded":"2023-10-18T16:07:41.433424+0000","last_fullsized":"2023-10-18T16:07:41.433424+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":8,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.801499+0000","last_change":"2023-10-18T16:07:41.801499+0000","last_active":"2023-10-18T16:07:41.801499+0000","last_peered":"2023-10-18T16:07:41.801499+0000","last_clean":"2023-10-18T16:07:41.801499+0000","last_became_active":"2023-10-18T16:07:41.800625+0000","last_became_peered":"2023-10-18T16:07:41.800625+0000","last_unstale":"2023-10-18T16:07:41.801499+0000","last_undegraded":"2023-10-18T16:07:41.801499+0000","last_fullsized":"2023-10-18T16:07:41.801499+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.789569+0000","last_change":"2023-10-18T16:07:41.435428+0000","last_active":"2023-10-18T16:07:41.789569+0000","last_peered":"2023-10-18T16:07:41.789569+0000","last_clean":"2023-10-18T16:07:41.789569+0000","last_became_active":"2023-10-18T16:07:41.435124+0000","last_became_peered":"2023-10-18T16:07:41.435124+0000","last_unstale":"2023-10-18T16:07:41.789569+0000","last_undegraded":"2023-10-18T16:07:41.789569+0000","last_fullsized":"2023-10-18T16:07:41.789569+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.416697+0000","last_change":"2023-10-18T16:07:36.542463+0000","last_active":"2023-10-18T16:07:41.416697+0000","last_peered":"2023-10-18T16:07:41.416697+0000","last_clean":"2023-10-18T16:07:41.416697+0000","last_became_active":"2023-10-18T16:07:36.542090+0000","last_became_peered":"2023-10-18T16:07:36.542090+0000","last_unstale":"2023-10-18T16:07:41.416697+0000","last_undegraded":"2023-10-18T16:07:41.416697+0000","last_fullsized":"2023-10-18T16:07:41.416697+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:35.522314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:35.522314+0000","last_clean_scrub_stamp":"2023-10-18T16:07:35.522314+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.807288+0000","last_change":"2023-10-18T16:07:41.802294+0000","last_active":"2023-10-18T16:07:41.807288+0000","last_peered":"2023-10-18T16:07:41.807288+0000","last_clean":"2023-10-18T16:07:41.807288+0000","last_became_active":"2023-10-18T16:07:41.801983+0000","last_became_peered":"2023-10-18T16:07:41.801983+0000","last_unstale":"2023-10-18T16:07:41.807288+0000","last_undegraded":"2023-10-18T16:07:41.807288+0000","last_fullsized":"2023-10-18T16:07:41.807288+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.428652+0000","last_change":"2023-10-18T16:07:36.540468+0000","last_active":"2023-10-18T16:07:41.428652+0000","last_peered":"2023-10-18T16:07:41.428652+0000","last_clean":"2023-10-18T16:07:41.428652+0000","last_became_active":"2023-10-18T16:07:36.540145+0000","last_became_peered":"2023-10-18T16:07:36.540145+0000","last_unstale":"2023-10-18T16:07:41.428652+0000","last_undegraded":"2023-10-18T16:07:41.428652+0000","last_fullsized":"2023-10-18T16:07:41.428652+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:35.522314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:35.522314+0000","last_clean_scrub_stamp":"2023-10-18T16:07:35.522314+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.435658+0000","last_change":"2023-10-18T16:07:41.431117+0000","last_active":"2023-10-18T16:07:41.435658+0000","last_peered":"2023-10-18T16:07:41.435658+0000","last_clean":"2023-10-18T16:07:41.435658+0000","last_became_active":"2023-10-18T16:07:41.430838+0000","last_became_peered":"2023-10-18T16:07:41.430838+0000","last_unstale":"2023-10-18T16:07:41.435658+0000","last_undegraded":"2023-10-18T16:07:41.435658+0000","last_fullsized":"2023-10-18T16:07:41.435658+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.806469+0000","last_change":"2023-10-18T16:07:41.803250+0000","last_active":"2023-10-18T16:07:41.806469+0000","last_peered":"2023-10-18T16:07:41.806469+0000","last_clean":"2023-10-18T16:07:41.806469+0000","last_became_active":"2023-10-18T16:07:41.802930+0000","last_became_peered":"2023-10-18T16:07:41.802930+0000","last_unstale":"2023-10-18T16:07:41.806469+0000","last_undegraded":"2023-10-18T16:07:41.806469+0000","last_fullsized":"2023-10-18T16:07:41.806469+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.807328+0000","last_change":"2023-10-18T16:07:41.803179+0000","last_active":"2023-10-18T16:07:41.807328+0000","last_peered":"2023-10-18T16:07:41.807328+0000","last_clean":"2023-10-18T16:07:41.807328+0000","last_became_active":"2023-10-18T16:07:41.802934+0000","last_became_peered":"2023-10-18T16:07:41.802934+0000","last_unstale":"2023-10-18T16:07:41.807328+0000","last_undegraded":"2023-10-18T16:07:41.807328+0000","last_fullsized":"2023-10-18T16:07:41.807328+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.444620+0000","last_change":"2023-10-18T16:07:41.443157+0000","last_active":"2023-10-18T16:07:41.444620+0000","last_peered":"2023-10-18T16:07:41.444620+0000","last_clean":"2023-10-18T16:07:41.444620+0000","last_became_active":"2023-10-18T16:07:41.442898+0000","last_became_peered":"2023-10-18T16:07:41.442898+0000","last_unstale":"2023-10-18T16:07:41.444620+0000","last_undegraded":"2023-10-18T16:07:41.444620+0000","last_fullsized":"2023-10-18T16:07:41.444620+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.807624+0000","last_change":"2023-10-18T16:07:41.804233+0000","last_active":"2023-10-18T16:07:41.807624+0000","last_peered":"2023-10-18T16:07:41.807624+0000","last_clean":"2023-10-18T16:07:41.807624+0000","last_became_active":"2023-10-18T16:07:41.803966+0000","last_became_peered":"2023-10-18T16:07:41.803966+0000","last_unstale":"2023-10-18T16:07:41.807624+0000","last_undegraded":"2023-10-18T16:07:41.807624+0000","last_fullsized":"2023-10-18T16:07:41.807624+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.447956+0000","last_change":"2023-10-18T16:07:41.443533+0000","last_active":"2023-10-18T16:07:41.447956+0000","last_peered":"2023-10-18T16:07:41.447956+0000","last_clean":"2023-10-18T16:07:41.447956+0000","last_became_active":"2023-10-18T16:07:41.443276+0000","last_became_peered":"2023-10-18T16:07:41.443276+0000","last_unstale":"2023-10-18T16:07:41.447956+0000","last_undegraded":"2023-10-18T16:07:41.447956+0000","last_fullsized":"2023-10-18T16:07:41.447956+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.805621+0000","last_change":"2023-10-18T16:07:41.799449+0000","last_active":"2023-10-18T16:07:41.805621+0000","last_peered":"2023-10-18T16:07:41.805621+0000","last_clean":"2023-10-18T16:07:41.805621+0000","last_became_active":"2023-10-18T16:07:41.799177+0000","last_became_peered":"2023-10-18T16:07:41.799177+0000","last_unstale":"2023-10-18T16:07:41.805621+0000","last_undegraded":"2023-10-18T16:07:41.805621+0000","last_fullsized":"2023-10-18T16:07:41.805621+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.448211+0000","last_change":"2023-10-18T16:07:41.443654+0000","last_active":"2023-10-18T16:07:41.448211+0000","last_peered":"2023-10-18T16:07:41.448211+0000","last_clean":"2023-10-18T16:07:41.448211+0000","last_became_active":"2023-10-18T16:07:41.443453+0000","last_became_peered":"2023-10-18T16:07:41.443453+0000","last_unstale":"2023-10-18T16:07:41.448211+0000","last_undegraded":"2023-10-18T16:07:41.448211+0000","last_fullsized":"2023-10-18T16:07:41.448211+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.802526+0000","last_change":"2023-10-18T16:07:41.798748+0000","last_active":"2023-10-18T16:07:41.802526+0000","last_peered":"2023-10-18T16:07:41.802526+0000","last_clean":"2023-10-18T16:07:41.802526+0000","last_became_active":"2023-10-18T16:07:41.798479+0000","last_became_peered":"2023-10-18T16:07:41.798479+0000","last_unstale":"2023-10-18T16:07:41.802526+0000","last_undegraded":"2023-10-18T16:07:41.802526+0000","last_fullsized":"2023-10-18T16:07:41.802526+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.788543+0000","last_change":"2023-10-18T16:07:41.438072+0000","last_active":"2023-10-18T16:07:41.788543+0000","last_peered":"2023-10-18T16:07:41.788543+0000","last_clean":"2023-10-18T16:07:41.788543+0000","last_became_active":"2023-10-18T16:07:41.437665+0000","last_became_peered":"2023-10-18T16:07:41.437665+0000","last_unstale":"2023-10-18T16:07:41.788543+0000","last_undegraded":"2023-10-18T16:07:41.788543+0000","last_fullsized":"2023-10-18T16:07:41.788543+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.806103+0000","last_change":"2023-10-18T16:07:41.800789+0000","last_active":"2023-10-18T16:07:41.806103+0000","last_peered":"2023-10-18T16:07:41.806103+0000","last_clean":"2023-10-18T16:07:41.806103+0000","last_became_active":"2023-10-18T16:07:41.800336+0000","last_became_peered":"2023-10-18T16:07:41.800336+0000","last_unstale":"2023-10-18T16:07:41.806103+0000","last_undegraded":"2023-10-18T16:07:41.806103+0000","last_fullsized":"2023-10-18T16:07:41.806103+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.440950+0000","last_change":"2023-10-18T16:07:41.439886+0000","last_active":"2023-10-18T16:07:41.440950+0000","last_peered":"2023-10-18T16:07:41.440950+0000","last_clean":"2023-10-18T16:07:41.440950+0000","last_became_active":"2023-10-18T16:07:41.439664+0000","last_became_peered":"2023-10-18T16:07:41.439664+0000","last_unstale":"2023-10-18T16:07:41.440950+0000","last_undegraded":"2023-10-18T16:07:41.440950+0000","last_fullsized":"2023-10-18T16:07:41.440950+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":8,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.797792+0000","last_change":"2023-10-18T16:07:41.797792+0000","last_active":"2023-10-18T16:07:41.797792+0000","last_peered":"2023-10-18T16:07:41.797792+0000","last_clean":"2023-10-18T16:07:41.797792+0000","last_became_active":"2023-10-18T16:07:41.797498+0000","last_became_peered":"2023-10-18T16:07:41.797498+0000","last_unstale":"2023-10-18T16:07:41.797792+0000","last_undegraded":"2023-10-18T16:07:41.797792+0000","last_fullsized":"2023-10-18T16:07:41.797792+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.439573+0000","last_change":"2023-10-18T16:07:41.434136+0000","last_active":"2023-10-18T16:07:41.439573+0000","last_peered":"2023-10-18T16:07:41.439573+0000","last_clean":"2023-10-18T16:07:41.439573+0000","last_became_active":"2023-10-18T16:07:41.433862+0000","last_became_peered":"2023-10-18T16:07:41.433862+0000","last_unstale":"2023-10-18T16:07:41.439573+0000","last_undegraded":"2023-10-18T16:07:41.439573+0000","last_fullsized":"2023-10-18T16:07:41.439573+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":8,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.797301+0000","last_change":"2023-10-18T16:07:41.797301+0000","last_active":"2023-10-18T16:07:41.797301+0000","last_peered":"2023-10-18T16:07:41.797301+0000","last_clean":"2023-10-18T16:07:41.797301+0000","last_became_active":"2023-10-18T16:07:41.797029+0000","last_became_peered":"2023-10-18T16:07:41.797029+0000","last_unstale":"2023-10-18T16:07:41.797301+0000","last_undegraded":"2023-10-18T16:07:41.797301+0000","last_fullsized":"2023-10-18T16:07:41.797301+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.795552+0000","last_change":"2023-10-18T16:07:41.426435+0000","last_active":"2023-10-18T16:07:41.795552+0000","last_peered":"2023-10-18T16:07:41.795552+0000","last_clean":"2023-10-18T16:07:41.795552+0000","last_became_active":"2023-10-18T16:07:41.426169+0000","last_became_peered":"2023-10-18T16:07:41.426169+0000","last_unstale":"2023-10-18T16:07:41.795552+0000","last_undegraded":"2023-10-18T16:07:41.795552+0000","last_fullsized":"2023-10-18T16:07:41.795552+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":8,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.808005+0000","last_change":"2023-10-18T16:07:41.808005+0000","last_active":"2023-10-18T16:07:41.808005+0000","last_peered":"2023-10-18T16:07:41.808005+0000","last_clean":"2023-10-18T16:07:41.808005+0000","last_became_active":"2023-10-18T16:07:41.807155+0000","last_became_peered":"2023-10-18T16:07:41.807155+0000","last_unstale":"2023-10-18T16:07:41.808005+0000","last_undegraded":"2023-10-18T16:07:41.808005+0000","last_fullsized":"2023-10-18T16:07:41.808005+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.446607+0000","last_change":"2023-10-18T16:07:41.437945+0000","last_active":"2023-10-18T16:07:41.446607+0000","last_peered":"2023-10-18T16:07:41.446607+0000","last_clean":"2023-10-18T16:07:41.446607+0000","last_became_active":"2023-10-18T16:07:41.437118+0000","last_became_peered":"2023-10-18T16:07:41.437118+0000","last_unstale":"2023-10-18T16:07:41.446607+0000","last_undegraded":"2023-10-18T16:07:41.446607+0000","last_fullsized":"2023-10-18T16:07:41.446607+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":8,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.801600+0000","last_change":"2023-10-18T16:07:41.801600+0000","last_active":"2023-10-18T16:07:41.801600+0000","last_peered":"2023-10-18T16:07:41.801600+0000","last_clean":"2023-10-18T16:07:41.801600+0000","last_became_active":"2023-10-18T16:07:41.801002+0000","last_became_peered":"2023-10-18T16:07:41.801002+0000","last_unstale":"2023-10-18T16:07:41.801600+0000","last_undegraded":"2023-10-18T16:07:41.801600+0000","last_fullsized":"2023-10-18T16:07:41.801600+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.435825+0000","last_change":"2023-10-18T16:07:41.432368+0000","last_active":"2023-10-18T16:07:41.435825+0000","last_peered":"2023-10-18T16:07:41.435825+0000","last_clean":"2023-10-18T16:07:41.435825+0000","last_became_active":"2023-10-18T16:07:41.431923+0000","last_became_peered":"2023-10-18T16:07:41.431923+0000","last_unstale":"2023-10-18T16:07:41.435825+0000","last_undegraded":"2023-10-18T16:07:41.435825+0000","last_fullsized":"2023-10-18T16:07:41.435825+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.810702+0000","last_change":"2023-10-18T16:07:41.809593+0000","last_active":"2023-10-18T16:07:41.810702+0000","last_peered":"2023-10-18T16:07:41.810702+0000","last_clean":"2023-10-18T16:07:41.810702+0000","last_became_active":"2023-10-18T16:07:41.809309+0000","last_became_peered":"2023-10-18T16:07:41.809309+0000","last_unstale":"2023-10-18T16:07:41.810702+0000","last_undegraded":"2023-10-18T16:07:41.810702+0000","last_fullsized":"2023-10-18T16:07:41.810702+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.789348+0000","last_change":"2023-10-18T16:07:41.443916+0000","last_active":"2023-10-18T16:07:41.789348+0000","last_peered":"2023-10-18T16:07:41.789348+0000","last_clean":"2023-10-18T16:07:41.789348+0000","last_became_active":"2023-10-18T16:07:41.443695+0000","last_became_peered":"2023-10-18T16:07:41.443695+0000","last_unstale":"2023-10-18T16:07:41.789348+0000","last_undegraded":"2023-10-18T16:07:41.789348+0000","last_fullsized":"2023-10-18T16:07:41.789348+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.806255+0000","last_change":"2023-10-18T16:07:41.803821+0000","last_active":"2023-10-18T16:07:41.806255+0000","last_peered":"2023-10-18T16:07:41.806255+0000","last_clean":"2023-10-18T16:07:41.806255+0000","last_became_active":"2023-10-18T16:07:41.803540+0000","last_became_peered":"2023-10-18T16:07:41.803540+0000","last_unstale":"2023-10-18T16:07:41.806255+0000","last_undegraded":"2023-10-18T16:07:41.806255+0000","last_fullsized":"2023-10-18T16:07:41.806255+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.433349+0000","last_change":"2023-10-18T16:07:41.429591+0000","last_active":"2023-10-18T16:07:41.433349+0000","last_peered":"2023-10-18T16:07:41.433349+0000","last_clean":"2023-10-18T16:07:41.433349+0000","last_became_active":"2023-10-18T16:07:41.429290+0000","last_became_peered":"2023-10-18T16:07:41.429290+0000","last_unstale":"2023-10-18T16:07:41.433349+0000","last_undegraded":"2023-10-18T16:07:41.433349+0000","last_fullsized":"2023-10-18T16:07:41.433349+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.799825+0000","last_change":"2023-10-18T16:07:41.795679+0000","last_active":"2023-10-18T16:07:41.799825+0000","last_peered":"2023-10-18T16:07:41.799825+0000","last_clean":"2023-10-18T16:07:41.799825+0000","last_became_active":"2023-10-18T16:07:41.795413+0000","last_became_peered":"2023-10-18T16:07:41.795413+0000","last_unstale":"2023-10-18T16:07:41.799825+0000","last_undegraded":"2023-10-18T16:07:41.799825+0000","last_fullsized":"2023-10-18T16:07:41.799825+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.444766+0000","last_change":"2023-10-18T16:07:41.436812+0000","last_active":"2023-10-18T16:07:41.444766+0000","last_peered":"2023-10-18T16:07:41.444766+0000","last_clean":"2023-10-18T16:07:41.444766+0000","last_became_active":"2023-10-18T16:07:41.436541+0000","last_became_peered":"2023-10-18T16:07:41.436541+0000","last_unstale":"2023-10-18T16:07:41.444766+0000","last_undegraded":"2023-10-18T16:07:41.444766+0000","last_fullsized":"2023-10-18T16:07:41.444766+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":8,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.803115+0000","last_change":"2023-10-18T16:07:41.803115+0000","last_active":"2023-10-18T16:07:41.803115+0000","last_peered":"2023-10-18T16:07:41.803115+0000","last_clean":"2023-10-18T16:07:41.803115+0000","last_became_active":"2023-10-18T16:07:41.802828+0000","last_became_peered":"2023-10-18T16:07:41.802828+0000","last_unstale":"2023-10-18T16:07:41.803115+0000","last_undegraded":"2023-10-18T16:07:41.803115+0000","last_fullsized":"2023-10-18T16:07:41.803115+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.789662+0000","last_change":"2023-10-18T16:07:41.443476+0000","last_active":"2023-10-18T16:07:41.789662+0000","last_peered":"2023-10-18T16:07:41.789662+0000","last_clean":"2023-10-18T16:07:41.789662+0000","last_became_active":"2023-10-18T16:07:41.443295+0000","last_became_peered":"2023-10-18T16:07:41.443295+0000","last_unstale":"2023-10-18T16:07:41.789662+0000","last_undegraded":"2023-10-18T16:07:41.789662+0000","last_fullsized":"2023-10-18T16:07:41.789662+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":8,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.800058+0000","last_change":"2023-10-18T16:07:41.800058+0000","last_active":"2023-10-18T16:07:41.800058+0000","last_peered":"2023-10-18T16:07:41.800058+0000","last_clean":"2023-10-18T16:07:41.800058+0000","last_became_active":"2023-10-18T16:07:41.799452+0000","last_became_peered":"2023-10-18T16:07:41.799452+0000","last_unstale":"2023-10-18T16:07:41.800058+0000","last_undegraded":"2023-10-18T16:07:41.800058+0000","last_fullsized":"2023-10-18T16:07:41.800058+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.446753+0000","last_change":"2023-10-18T16:07:41.440724+0000","last_active":"2023-10-18T16:07:41.446753+0000","last_peered":"2023-10-18T16:07:41.446753+0000","last_clean":"2023-10-18T16:07:41.446753+0000","last_became_active":"2023-10-18T16:07:41.440435+0000","last_became_peered":"2023-10-18T16:07:41.440435+0000","last_unstale":"2023-10-18T16:07:41.446753+0000","last_undegraded":"2023-10-18T16:07:41.446753+0000","last_fullsized":"2023-10-18T16:07:41.446753+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.806559+0000","last_change":"2023-10-18T16:07:41.804438+0000","last_active":"2023-10-18T16:07:41.806559+0000","last_peered":"2023-10-18T16:07:41.806559+0000","last_clean":"2023-10-18T16:07:41.806559+0000","last_became_active":"2023-10-18T16:07:41.804126+0000","last_became_peered":"2023-10-18T16:07:41.804126+0000","last_unstale":"2023-10-18T16:07:41.806559+0000","last_undegraded":"2023-10-18T16:07:41.806559+0000","last_fullsized":"2023-10-18T16:07:41.806559+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.788864+0000","last_change":"2023-10-18T16:07:41.447812+0000","last_active":"2023-10-18T16:07:41.788864+0000","last_peered":"2023-10-18T16:07:41.788864+0000","last_clean":"2023-10-18T16:07:41.788864+0000","last_became_active":"2023-10-18T16:07:41.447479+0000","last_became_peered":"2023-10-18T16:07:41.447479+0000","last_unstale":"2023-10-18T16:07:41.788864+0000","last_undegraded":"2023-10-18T16:07:41.788864+0000","last_fullsized":"2023-10-18T16:07:41.788864+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.804861+0000","last_change":"2023-10-18T16:07:41.801084+0000","last_active":"2023-10-18T16:07:41.804861+0000","last_peered":"2023-10-18T16:07:41.804861+0000","last_clean":"2023-10-18T16:07:41.804861+0000","last_became_active":"2023-10-18T16:07:41.800783+0000","last_became_peered":"2023-10-18T16:07:41.800783+0000","last_unstale":"2023-10-18T16:07:41.804861+0000","last_undegraded":"2023-10-18T16:07:41.804861+0000","last_fullsized":"2023-10-18T16:07:41.804861+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.446593+0000","last_change":"2023-10-18T16:07:41.437755+0000","last_active":"2023-10-18T16:07:41.446593+0000","last_peered":"2023-10-18T16:07:41.446593+0000","last_clean":"2023-10-18T16:07:41.446593+0000","last_became_active":"2023-10-18T16:07:41.436781+0000","last_became_peered":"2023-10-18T16:07:41.436781+0000","last_unstale":"2023-10-18T16:07:41.446593+0000","last_undegraded":"2023-10-18T16:07:41.446593+0000","last_fullsized":"2023-10-18T16:07:41.446593+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.810938+0000","last_change":"2023-10-18T16:07:41.808318+0000","last_active":"2023-10-18T16:07:41.810938+0000","last_peered":"2023-10-18T16:07:41.810938+0000","last_clean":"2023-10-18T16:07:41.810938+0000","last_became_active":"2023-10-18T16:07:41.807983+0000","last_became_peered":"2023-10-18T16:07:41.807983+0000","last_unstale":"2023-10-18T16:07:41.810938+0000","last_undegraded":"2023-10-18T16:07:41.810938+0000","last_fullsized":"2023-10-18T16:07:41.810938+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.443251+0000","last_change":"2023-10-18T16:07:41.442119+0000","last_active":"2023-10-18T16:07:41.443251+0000","last_peered":"2023-10-18T16:07:41.443251+0000","last_clean":"2023-10-18T16:07:41.443251+0000","last_became_active":"2023-10-18T16:07:41.441816+0000","last_became_peered":"2023-10-18T16:07:41.441816+0000","last_unstale":"2023-10-18T16:07:41.443251+0000","last_undegraded":"2023-10-18T16:07:41.443251+0000","last_fullsized":"2023-10-18T16:07:41.443251+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.810523+0000","last_change":"2023-10-18T16:07:41.809378+0000","last_active":"2023-10-18T16:07:41.810523+0000","last_peered":"2023-10-18T16:07:41.810523+0000","last_clean":"2023-10-18T16:07:41.810523+0000","last_became_active":"2023-10-18T16:07:41.809102+0000","last_became_peered":"2023-10-18T16:07:41.809102+0000","last_unstale":"2023-10-18T16:07:41.810523+0000","last_undegraded":"2023-10-18T16:07:41.810523+0000","last_fullsized":"2023-10-18T16:07:41.810523+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.447410+0000","last_change":"2023-10-18T16:07:41.442792+0000","last_active":"2023-10-18T16:07:41.447410+0000","last_peered":"2023-10-18T16:07:41.447410+0000","last_clean":"2023-10-18T16:07:41.447410+0000","last_became_active":"2023-10-18T16:07:41.442506+0000","last_became_peered":"2023-10-18T16:07:41.442506+0000","last_unstale":"2023-10-18T16:07:41.447410+0000","last_undegraded":"2023-10-18T16:07:41.447410+0000","last_fullsized":"2023-10-18T16:07:41.447410+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":8,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.804704+0000","last_change":"2023-10-18T16:07:41.804704+0000","last_active":"2023-10-18T16:07:41.804704+0000","last_peered":"2023-10-18T16:07:41.804704+0000","last_clean":"2023-10-18T16:07:41.804704+0000","last_became_active":"2023-10-18T16:07:41.804437+0000","last_became_peered":"2023-10-18T16:07:41.804437+0000","last_unstale":"2023-10-18T16:07:41.804704+0000","last_undegraded":"2023-10-18T16:07:41.804704+0000","last_fullsized":"2023-10-18T16:07:41.804704+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.798211+0000","last_change":"2023-10-18T16:07:41.436171+0000","last_active":"2023-10-18T16:07:41.798211+0000","last_peered":"2023-10-18T16:07:41.798211+0000","last_clean":"2023-10-18T16:07:41.798211+0000","last_became_active":"2023-10-18T16:07:41.435844+0000","last_became_peered":"2023-10-18T16:07:41.435844+0000","last_unstale":"2023-10-18T16:07:41.798211+0000","last_undegraded":"2023-10-18T16:07:41.798211+0000","last_fullsized":"2023-10-18T16:07:41.798211+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.802055+0000","last_change":"2023-10-18T16:07:41.796556+0000","last_active":"2023-10-18T16:07:41.802055+0000","last_peered":"2023-10-18T16:07:41.802055+0000","last_clean":"2023-10-18T16:07:41.802055+0000","last_became_active":"2023-10-18T16:07:41.796291+0000","last_became_peered":"2023-10-18T16:07:41.796291+0000","last_unstale":"2023-10-18T16:07:41.802055+0000","last_undegraded":"2023-10-18T16:07:41.802055+0000","last_fullsized":"2023-10-18T16:07:41.802055+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.795432+0000","last_change":"2023-10-18T16:07:41.434332+0000","last_active":"2023-10-18T16:07:41.795432+0000","last_peered":"2023-10-18T16:07:41.795432+0000","last_clean":"2023-10-18T16:07:41.795432+0000","last_became_active":"2023-10-18T16:07:41.434076+0000","last_became_peered":"2023-10-18T16:07:41.434076+0000","last_unstale":"2023-10-18T16:07:41.795432+0000","last_undegraded":"2023-10-18T16:07:41.795432+0000","last_fullsized":"2023-10-18T16:07:41.795432+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.808487+0000","last_change":"2023-10-18T16:07:41.806379+0000","last_active":"2023-10-18T16:07:41.808487+0000","last_peered":"2023-10-18T16:07:41.808487+0000","last_clean":"2023-10-18T16:07:41.808487+0000","last_became_active":"2023-10-18T16:07:41.806018+0000","last_became_peered":"2023-10-18T16:07:41.806018+0000","last_unstale":"2023-10-18T16:07:41.808487+0000","last_undegraded":"2023-10-18T16:07:41.808487+0000","last_fullsized":"2023-10-18T16:07:41.808487+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.446654+0000","last_change":"2023-10-18T16:07:41.438688+0000","last_active":"2023-10-18T16:07:41.446654+0000","last_peered":"2023-10-18T16:07:41.446654+0000","last_clean":"2023-10-18T16:07:41.446654+0000","last_became_active":"2023-10-18T16:07:41.438405+0000","last_became_peered":"2023-10-18T16:07:41.438405+0000","last_unstale":"2023-10-18T16:07:41.446654+0000","last_undegraded":"2023-10-18T16:07:41.446654+0000","last_fullsized":"2023-10-18T16:07:41.446654+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":8,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.800230+0000","last_change":"2023-10-18T16:07:41.800230+0000","last_active":"2023-10-18T16:07:41.800230+0000","last_peered":"2023-10-18T16:07:41.800230+0000","last_clean":"2023-10-18T16:07:41.800230+0000","last_became_active":"2023-10-18T16:07:41.799787+0000","last_became_peered":"2023-10-18T16:07:41.799787+0000","last_unstale":"2023-10-18T16:07:41.800230+0000","last_undegraded":"2023-10-18T16:07:41.800230+0000","last_fullsized":"2023-10-18T16:07:41.800230+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.446769+0000","last_change":"2023-10-18T16:07:41.440794+0000","last_active":"2023-10-18T16:07:41.446769+0000","last_peered":"2023-10-18T16:07:41.446769+0000","last_clean":"2023-10-18T16:07:41.446769+0000","last_became_active":"2023-10-18T16:07:41.440412+0000","last_became_peered":"2023-10-18T16:07:41.440412+0000","last_unstale":"2023-10-18T16:07:41.446769+0000","last_undegraded":"2023-10-18T16:07:41.446769+0000","last_fullsized":"2023-10-18T16:07:41.446769+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":8,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.801402+0000","last_change":"2023-10-18T16:07:41.801402+0000","last_active":"2023-10-18T16:07:41.801402+0000","last_peered":"2023-10-18T16:07:41.801402+0000","last_clean":"2023-10-18T16:07:41.801402+0000","last_became_active":"2023-10-18T16:07:41.800562+0000","last_became_peered":"2023-10-18T16:07:41.800562+0000","last_unstale":"2023-10-18T16:07:41.801402+0000","last_undegraded":"2023-10-18T16:07:41.801402+0000","last_fullsized":"2023-10-18T16:07:41.801402+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.446301+0000","last_change":"2023-10-18T16:07:41.445361+0000","last_active":"2023-10-18T16:07:41.446301+0000","last_peered":"2023-10-18T16:07:41.446301+0000","last_clean":"2023-10-18T16:07:41.446301+0000","last_became_active":"2023-10-18T16:07:41.445159+0000","last_became_peered":"2023-10-18T16:07:41.445159+0000","last_unstale":"2023-10-18T16:07:41.446301+0000","last_undegraded":"2023-10-18T16:07:41.446301+0000","last_fullsized":"2023-10-18T16:07:41.446301+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.806606+0000","last_change":"2023-10-18T16:07:41.806046+0000","last_active":"2023-10-18T16:07:41.806606+0000","last_peered":"2023-10-18T16:07:41.806606+0000","last_clean":"2023-10-18T16:07:41.806606+0000","last_became_active":"2023-10-18T16:07:41.805799+0000","last_became_peered":"2023-10-18T16:07:41.805799+0000","last_unstale":"2023-10-18T16:07:41.806606+0000","last_undegraded":"2023-10-18T16:07:41.806606+0000","last_fullsized":"2023-10-18T16:07:41.806606+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.430087+0000","last_change":"2023-10-18T16:07:41.425946+0000","last_active":"2023-10-18T16:07:41.430087+0000","last_peered":"2023-10-18T16:07:41.430087+0000","last_clean":"2023-10-18T16:07:41.430087+0000","last_became_active":"2023-10-18T16:07:41.425673+0000","last_became_peered":"2023-10-18T16:07:41.425673+0000","last_unstale":"2023-10-18T16:07:41.430087+0000","last_undegraded":"2023-10-18T16:07:41.430087+0000","last_fullsized":"2023-10-18T16:07:41.430087+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.811135+0000","last_change":"2023-10-18T16:07:41.810501+0000","last_active":"2023-10-18T16:07:41.811135+0000","last_peered":"2023-10-18T16:07:41.811135+0000","last_clean":"2023-10-18T16:07:41.811135+0000","last_became_active":"2023-10-18T16:07:41.810233+0000","last_became_peered":"2023-10-18T16:07:41.810233+0000","last_unstale":"2023-10-18T16:07:41.811135+0000","last_undegraded":"2023-10-18T16:07:41.811135+0000","last_fullsized":"2023-10-18T16:07:41.811135+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.788717+0000","last_change":"2023-10-18T16:07:41.434399+0000","last_active":"2023-10-18T16:07:41.788717+0000","last_peered":"2023-10-18T16:07:41.788717+0000","last_clean":"2023-10-18T16:07:41.788717+0000","last_became_active":"2023-10-18T16:07:41.434143+0000","last_became_peered":"2023-10-18T16:07:41.434143+0000","last_unstale":"2023-10-18T16:07:41.788717+0000","last_undegraded":"2023-10-18T16:07:41.788717+0000","last_fullsized":"2023-10-18T16:07:41.788717+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.801283+0000","last_change":"2023-10-18T16:07:41.799082+0000","last_active":"2023-10-18T16:07:41.801283+0000","last_peered":"2023-10-18T16:07:41.801283+0000","last_clean":"2023-10-18T16:07:41.801283+0000","last_became_active":"2023-10-18T16:07:41.798807+0000","last_became_peered":"2023-10-18T16:07:41.798807+0000","last_unstale":"2023-10-18T16:07:41.801283+0000","last_undegraded":"2023-10-18T16:07:41.801283+0000","last_fullsized":"2023-10-18T16:07:41.801283+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.796407+0000","last_change":"2023-10-18T16:07:41.445579+0000","last_active":"2023-10-18T16:07:41.796407+0000","last_peered":"2023-10-18T16:07:41.796407+0000","last_clean":"2023-10-18T16:07:41.796407+0000","last_became_active":"2023-10-18T16:07:41.445348+0000","last_became_peered":"2023-10-18T16:07:41.445348+0000","last_unstale":"2023-10-18T16:07:41.796407+0000","last_undegraded":"2023-10-18T16:07:41.796407+0000","last_fullsized":"2023-10-18T16:07:41.796407+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":8,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.802347+0000","last_change":"2023-10-18T16:07:41.802347+0000","last_active":"2023-10-18T16:07:41.802347+0000","last_peered":"2023-10-18T16:07:41.802347+0000","last_clean":"2023-10-18T16:07:41.802347+0000","last_became_active":"2023-10-18T16:07:41.802061+0000","last_became_peered":"2023-10-18T16:07:41.802061+0000","last_unstale":"2023-10-18T16:07:41.802347+0000","last_undegraded":"2023-10-18T16:07:41.802347+0000","last_fullsized":"2023-10-18T16:07:41.802347+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.447357+0000","last_change":"2023-10-18T16:07:41.439414+0000","last_active":"2023-10-18T16:07:41.447357+0000","last_peered":"2023-10-18T16:07:41.447357+0000","last_clean":"2023-10-18T16:07:41.447357+0000","last_became_active":"2023-10-18T16:07:41.439137+0000","last_became_peered":"2023-10-18T16:07:41.439137+0000","last_unstale":"2023-10-18T16:07:41.447357+0000","last_undegraded":"2023-10-18T16:07:41.447357+0000","last_fullsized":"2023-10-18T16:07:41.447357+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.810429+0000","last_change":"2023-10-18T16:07:41.808426+0000","last_active":"2023-10-18T16:07:41.810429+0000","last_peered":"2023-10-18T16:07:41.810429+0000","last_clean":"2023-10-18T16:07:41.810429+0000","last_became_active":"2023-10-18T16:07:41.808146+0000","last_became_peered":"2023-10-18T16:07:41.808146+0000","last_unstale":"2023-10-18T16:07:41.810429+0000","last_undegraded":"2023-10-18T16:07:41.810429+0000","last_fullsized":"2023-10-18T16:07:41.810429+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.447293+0000","last_change":"2023-10-18T16:07:41.439810+0000","last_active":"2023-10-18T16:07:41.447293+0000","last_peered":"2023-10-18T16:07:41.447293+0000","last_clean":"2023-10-18T16:07:41.447293+0000","last_became_active":"2023-10-18T16:07:41.439545+0000","last_became_peered":"2023-10-18T16:07:41.439545+0000","last_unstale":"2023-10-18T16:07:41.447293+0000","last_undegraded":"2023-10-18T16:07:41.447293+0000","last_fullsized":"2023-10-18T16:07:41.447293+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.806045+0000","last_change":"2023-10-18T16:07:41.800617+0000","last_active":"2023-10-18T16:07:41.806045+0000","last_peered":"2023-10-18T16:07:41.806045+0000","last_clean":"2023-10-18T16:07:41.806045+0000","last_became_active":"2023-10-18T16:07:41.799998+0000","last_became_peered":"2023-10-18T16:07:41.799998+0000","last_unstale":"2023-10-18T16:07:41.806045+0000","last_undegraded":"2023-10-18T16:07:41.806045+0000","last_fullsized":"2023-10-18T16:07:41.806045+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.798396+0000","last_change":"2023-10-18T16:07:41.442796+0000","last_active":"2023-10-18T16:07:41.798396+0000","last_peered":"2023-10-18T16:07:41.798396+0000","last_clean":"2023-10-18T16:07:41.798396+0000","last_became_active":"2023-10-18T16:07:41.442007+0000","last_became_peered":"2023-10-18T16:07:41.442007+0000","last_unstale":"2023-10-18T16:07:41.798396+0000","last_undegraded":"2023-10-18T16:07:41.798396+0000","last_fullsized":"2023-10-18T16:07:41.798396+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3f","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.801310+0000","last_change":"2023-10-18T16:07:41.799046+0000","last_active":"2023-10-18T16:07:41.801310+0000","last_peered":"2023-10-18T16:07:41.801310+0000","last_clean":"2023-10-18T16:07:41.801310+0000","last_became_active":"2023-10-18T16:07:41.798754+0000","last_became_peered":"2023-10-18T16:07:41.798754+0000","last_unstale":"2023-10-18T16:07:41.801310+0000","last_undegraded":"2023-10-18T16:07:41.801310+0000","last_fullsized":"2023-10-18T16:07:41.801310+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.789919+0000","last_change":"2023-10-18T16:07:41.441443+0000","last_active":"2023-10-18T16:07:41.789919+0000","last_peered":"2023-10-18T16:07:41.789919+0000","last_clean":"2023-10-18T16:07:41.789919+0000","last_became_active":"2023-10-18T16:07:41.441208+0000","last_became_peered":"2023-10-18T16:07:41.441208+0000","last_unstale":"2023-10-18T16:07:41.789919+0000","last_undegraded":"2023-10-18T16:07:41.789919+0000","last_fullsized":"2023-10-18T16:07:41.789919+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":8,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.800460+0000","last_change":"2023-10-18T16:07:41.800460+0000","last_active":"2023-10-18T16:07:41.800460+0000","last_peered":"2023-10-18T16:07:41.800460+0000","last_clean":"2023-10-18T16:07:41.800460+0000","last_became_active":"2023-10-18T16:07:41.800013+0000","last_became_peered":"2023-10-18T16:07:41.800013+0000","last_unstale":"2023-10-18T16:07:41.800460+0000","last_undegraded":"2023-10-18T16:07:41.800460+0000","last_fullsized":"2023-10-18T16:07:41.800460+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.799786+0000","last_change":"2023-10-18T16:07:41.438711+0000","last_active":"2023-10-18T16:07:41.799786+0000","last_peered":"2023-10-18T16:07:41.799786+0000","last_clean":"2023-10-18T16:07:41.799786+0000","last_became_active":"2023-10-18T16:07:41.438408+0000","last_became_peered":"2023-10-18T16:07:41.438408+0000","last_unstale":"2023-10-18T16:07:41.799786+0000","last_undegraded":"2023-10-18T16:07:41.799786+0000","last_fullsized":"2023-10-18T16:07:41.799786+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.806161+0000","last_change":"2023-10-18T16:07:41.800718+0000","last_active":"2023-10-18T16:07:41.806161+0000","last_peered":"2023-10-18T16:07:41.806161+0000","last_clean":"2023-10-18T16:07:41.806161+0000","last_became_active":"2023-10-18T16:07:41.800400+0000","last_became_peered":"2023-10-18T16:07:41.800400+0000","last_unstale":"2023-10-18T16:07:41.806161+0000","last_undegraded":"2023-10-18T16:07:41.806161+0000","last_fullsized":"2023-10-18T16:07:41.806161+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.446623+0000","last_change":"2023-10-18T16:07:41.438151+0000","last_active":"2023-10-18T16:07:41.446623+0000","last_peered":"2023-10-18T16:07:41.446623+0000","last_clean":"2023-10-18T16:07:41.446623+0000","last_became_active":"2023-10-18T16:07:41.437504+0000","last_became_peered":"2023-10-18T16:07:41.437504+0000","last_unstale":"2023-10-18T16:07:41.446623+0000","last_undegraded":"2023-10-18T16:07:41.446623+0000","last_fullsized":"2023-10-18T16:07:41.446623+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":8,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.802411+0000","last_change":"2023-10-18T16:07:41.802411+0000","last_active":"2023-10-18T16:07:41.802411+0000","last_peered":"2023-10-18T16:07:41.802411+0000","last_clean":"2023-10-18T16:07:41.802411+0000","last_became_active":"2023-10-18T16:07:41.802124+0000","last_became_peered":"2023-10-18T16:07:41.802124+0000","last_unstale":"2023-10-18T16:07:41.802411+0000","last_undegraded":"2023-10-18T16:07:41.802411+0000","last_fullsized":"2023-10-18T16:07:41.802411+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.797104+0000","last_change":"2023-10-18T16:07:41.439071+0000","last_active":"2023-10-18T16:07:41.797104+0000","last_peered":"2023-10-18T16:07:41.797104+0000","last_clean":"2023-10-18T16:07:41.797104+0000","last_became_active":"2023-10-18T16:07:41.438144+0000","last_became_peered":"2023-10-18T16:07:41.438144+0000","last_unstale":"2023-10-18T16:07:41.797104+0000","last_undegraded":"2023-10-18T16:07:41.797104+0000","last_fullsized":"2023-10-18T16:07:41.797104+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":8,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.798403+0000","last_change":"2023-10-18T16:07:41.798403+0000","last_active":"2023-10-18T16:07:41.798403+0000","last_peered":"2023-10-18T16:07:41.798403+0000","last_clean":"2023-10-18T16:07:41.798403+0000","last_became_active":"2023-10-18T16:07:41.798149+0000","last_became_peered":"2023-10-18T16:07:41.798149+0000","last_unstale":"2023-10-18T16:07:41.798403+0000","last_undegraded":"2023-10-18T16:07:41.798403+0000","last_fullsized":"2023-10-18T16:07:41.798403+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.438801+0000","last_change":"2023-10-18T16:07:41.432914+0000","last_active":"2023-10-18T16:07:41.438801+0000","last_peered":"2023-10-18T16:07:41.438801+0000","last_clean":"2023-10-18T16:07:41.438801+0000","last_became_active":"2023-10-18T16:07:41.432601+0000","last_became_peered":"2023-10-18T16:07:41.432601+0000","last_unstale":"2023-10-18T16:07:41.438801+0000","last_undegraded":"2023-10-18T16:07:41.438801+0000","last_fullsized":"2023-10-18T16:07:41.438801+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.806049+0000","last_change":"2023-10-18T16:07:41.801443+0000","last_active":"2023-10-18T16:07:41.806049+0000","last_peered":"2023-10-18T16:07:41.806049+0000","last_clean":"2023-10-18T16:07:41.806049+0000","last_became_active":"2023-10-18T16:07:41.801271+0000","last_became_peered":"2023-10-18T16:07:41.801271+0000","last_unstale":"2023-10-18T16:07:41.806049+0000","last_undegraded":"2023-10-18T16:07:41.806049+0000","last_fullsized":"2023-10-18T16:07:41.806049+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.436412+0000","last_change":"2023-10-18T16:07:41.434525+0000","last_active":"2023-10-18T16:07:41.436412+0000","last_peered":"2023-10-18T16:07:41.436412+0000","last_clean":"2023-10-18T16:07:41.436412+0000","last_became_active":"2023-10-18T16:07:41.434220+0000","last_became_peered":"2023-10-18T16:07:41.434220+0000","last_unstale":"2023-10-18T16:07:41.436412+0000","last_undegraded":"2023-10-18T16:07:41.436412+0000","last_fullsized":"2023-10-18T16:07:41.436412+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.808917+0000","last_change":"2023-10-18T16:07:41.807740+0000","last_active":"2023-10-18T16:07:41.808917+0000","last_peered":"2023-10-18T16:07:41.808917+0000","last_clean":"2023-10-18T16:07:41.808917+0000","last_became_active":"2023-10-18T16:07:41.807473+0000","last_became_peered":"2023-10-18T16:07:41.807473+0000","last_unstale":"2023-10-18T16:07:41.808917+0000","last_undegraded":"2023-10-18T16:07:41.808917+0000","last_fullsized":"2023-10-18T16:07:41.808917+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.433447+0000","last_change":"2023-10-18T16:07:41.427326+0000","last_active":"2023-10-18T16:07:41.433447+0000","last_peered":"2023-10-18T16:07:41.433447+0000","last_clean":"2023-10-18T16:07:41.433447+0000","last_became_active":"2023-10-18T16:07:41.427083+0000","last_became_peered":"2023-10-18T16:07:41.427083+0000","last_unstale":"2023-10-18T16:07:41.433447+0000","last_undegraded":"2023-10-18T16:07:41.433447+0000","last_fullsized":"2023-10-18T16:07:41.433447+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":8,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.804453+0000","last_change":"2023-10-18T16:07:41.804453+0000","last_active":"2023-10-18T16:07:41.804453+0000","last_peered":"2023-10-18T16:07:41.804453+0000","last_clean":"2023-10-18T16:07:41.804453+0000","last_became_active":"2023-10-18T16:07:41.802577+0000","last_became_peered":"2023-10-18T16:07:41.802577+0000","last_unstale":"2023-10-18T16:07:41.804453+0000","last_undegraded":"2023-10-18T16:07:41.804453+0000","last_fullsized":"2023-10-18T16:07:41.804453+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.798474+0000","last_change":"2023-10-18T16:07:41.436757+0000","last_active":"2023-10-18T16:07:41.798474+0000","last_peered":"2023-10-18T16:07:41.798474+0000","last_clean":"2023-10-18T16:07:41.798474+0000","last_became_active":"2023-10-18T16:07:41.436423+0000","last_became_peered":"2023-10-18T16:07:41.436423+0000","last_unstale":"2023-10-18T16:07:41.798474+0000","last_undegraded":"2023-10-18T16:07:41.798474+0000","last_fullsized":"2023-10-18T16:07:41.798474+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574851,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297320,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074520,"statfs":{"total":96636764160,"available":96332308480,"internally_reserved":0,"allocated":229376,"data_stored":85341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53700000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52400000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68899999999999995}]}]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297308,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074532,"statfs":{"total":96636764160,"available":96332320768,"internally_reserved":0,"allocated":217088,"data_stored":78637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66300000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65000000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51300000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574851,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297308,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074532,"statfs":{"total":96636764160,"available":96332320768,"internally_reserved":0,"allocated":217088,"data_stored":78637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.318}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.373}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574851,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297308,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074532,"statfs":{"total":96636764160,"available":96332320768,"internally_reserved":0,"allocated":217088,"data_stored":78637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28899999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65900000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56799999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60899999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574851,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297320,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074520,"statfs":{"total":96636764160,"available":96332308480,"internally_reserved":0,"allocated":229376,"data_stored":85341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34200000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36799999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47999999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66300000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63700000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574852,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297308,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074532,"statfs":{"total":96636764160,"available":96332320768,"internally_reserved":0,"allocated":217088,"data_stored":78637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34599999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35899999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58599999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68999999999999995}]}]},{"osd":4,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297320,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074520,"statfs":{"total":96636764160,"available":96332308480,"internally_reserved":0,"allocated":229376,"data_stored":85341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58699999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69099999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42799999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38500000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]}]},{"osd":5,"up_from":13,"seq":55834574851,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297308,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074532,"statfs":{"total":96636764160,"available":96332320768,"internally_reserved":0,"allocated":217088,"data_stored":78637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70299999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68999999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53300000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-18T16:07:47.414 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-18T16:07:47.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.554+0000 7f214ec4a700 1 Processor -- start 2023-10-18T16:07:47.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.554+0000 7f214ec4a700 1 -- start start 2023-10-18T16:07:47.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.555+0000 7f214ec4a700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f214814d7c0 0x7f21480a8de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:47.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.555+0000 7f214ec4a700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f214814dba0 0x7f2148151cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:47.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.555+0000 7f214ec4a700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2148151840 con 0x7f2148064360 2023-10-18T16:07:47.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.555+0000 7f214ec4a700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21480c42d0 con 0x7f214814dba0 2023-10-18T16:07:47.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.555+0000 7f214ec4a700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2148064830 con 0x7f214814d7c0 2023-10-18T16:07:47.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.555+0000 7f214e449700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f214814dba0 0x7f2148151cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:47.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.555+0000 7f214e449700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f214814dba0 0x7f2148151cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44088/0 (socket says 172.21.15.88:44088) 2023-10-18T16:07:47.562 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.555+0000 7f214e449700 1 -- 172.21.15.88:0/1557041663 learned_addr learned my addr 172.21.15.88:0/1557041663 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:47.562 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.555+0000 7f214d447700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f2148064360 0x7f21481538a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:52422/0 (socket says 172.21.15.88:52422) 2023-10-18T16:07:47.562 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.555+0000 7f214dc48700 1 --2- 172.21.15.88:0/1557041663 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f214814d7c0 0x7f21480a8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:47.562 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.555+0000 7f214e449700 1 -- 172.21.15.88:0/1557041663 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f214814d7c0 msgr2=0x7f21480a8de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:47.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.555+0000 7f214e449700 1 --2- 172.21.15.88:0/1557041663 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f214814d7c0 0x7f21480a8de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:47.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.555+0000 7f214e449700 1 -- 172.21.15.88:0/1557041663 >> v1:172.21.15.88:6789/0 conn(0x7f2148064360 legacy=0x7f21481538a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:47.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.555+0000 7f214cc46700 1 -- 172.21.15.88:0/1557041663 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3049804052 0 0) 0x7f2148151840 con 0x7f2148064360 2023-10-18T16:07:47.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.555+0000 7f214e449700 1 -- 172.21.15.88:0/1557041663 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2148064b10 con 0x7f214814dba0 2023-10-18T16:07:47.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.555+0000 7f214e449700 1 --2- 172.21.15.88:0/1557041663 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f214814dba0 0x7f2148151cf0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f2140009f20 tx=0x7f2140009840).ready entity=mon.0 client_cookie=eeedfd6c597d8e15 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:47.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.555+0000 7f214cc46700 1 -- 172.21.15.88:0/1557041663 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f214000e070 con 0x7f214814dba0 2023-10-18T16:07:47.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.555+0000 7f214cc46700 1 -- 172.21.15.88:0/1557041663 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f214000ba80 con 0x7f214814dba0 2023-10-18T16:07:47.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.555+0000 7f214cc46700 1 -- 172.21.15.88:0/1557041663 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f214000be00 con 0x7f214814dba0 2023-10-18T16:07:47.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.556+0000 7f214ec4a700 1 -- 172.21.15.88:0/1557041663 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f214814dba0 msgr2=0x7f2148151cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:47.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.556+0000 7f214ec4a700 1 --2- 172.21.15.88:0/1557041663 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f214814dba0 0x7f2148151cf0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f2140009f20 tx=0x7f2140009840).stop 2023-10-18T16:07:47.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.556+0000 7f214ec4a700 1 -- 172.21.15.88:0/1557041663 shutdown_connections 2023-10-18T16:07:47.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.556+0000 7f214ec4a700 1 --2- 172.21.15.88:0/1557041663 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f214814dba0 0x7f2148151cf0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:47.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.556+0000 7f214ec4a700 1 --2- 172.21.15.88:0/1557041663 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f214814d7c0 0x7f21480a8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:47.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.556+0000 7f214ec4a700 1 -- 172.21.15.88:0/1557041663 >> 172.21.15.88:0/1557041663 conn(0x7f2148069d80 msgr2=0x7f214806a190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:47.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.556+0000 7f214ec4a700 1 -- 172.21.15.88:0/1557041663 shutdown_connections 2023-10-18T16:07:47.566 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.556+0000 7f214ec4a700 1 -- 172.21.15.88:0/1557041663 wait complete. 2023-10-18T16:07:47.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.557+0000 7f214ec4a700 1 Processor -- start 2023-10-18T16:07:47.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.557+0000 7f214ec4a700 1 -- start start 2023-10-18T16:07:47.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.558+0000 7f214ec4a700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2148064360 0x7f2148150d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:47.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.558+0000 7f214ec4a700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f214814d7c0 0x7f21480ab250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:47.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.558+0000 7f214ec4a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f214814dba0 0x7f21480ab740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:47.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.558+0000 7f214ec4a700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2148150380 con 0x7f214814d7c0 2023-10-18T16:07:47.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.558+0000 7f214ec4a700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f214814d610 con 0x7f214814dba0 2023-10-18T16:07:47.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.558+0000 7f214ec4a700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f214814f370 con 0x7f2148064360 2023-10-18T16:07:47.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.558+0000 7f214d447700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f214814d7c0 0x7f21480ab250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:47.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.558+0000 7f214d447700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f214814d7c0 0x7f21480ab250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44092/0 (socket says 172.21.15.88:44092) 2023-10-18T16:07:47.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.558+0000 7f214d447700 1 -- 172.21.15.88:0/2475980124 learned_addr learned my addr 172.21.15.88:0/2475980124 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:47.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.558+0000 7f214dc48700 1 --2- 172.21.15.88:0/2475980124 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2148064360 0x7f2148150d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:47.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.558+0000 7f214e449700 1 --2- 172.21.15.88:0/2475980124 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f214814dba0 0x7f21480ab740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:47.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.558+0000 7f214d447700 1 -- 172.21.15.88:0/2475980124 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2148064360 msgr2=0x7f2148150d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:47.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.558+0000 7f214d447700 1 --2- 172.21.15.88:0/2475980124 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2148064360 0x7f2148150d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:47.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.558+0000 7f214d447700 1 -- 172.21.15.88:0/2475980124 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f214814dba0 msgr2=0x7f21480ab740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:47.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.558+0000 7f214d447700 1 --2- 172.21.15.88:0/2475980124 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f214814dba0 0x7f21480ab740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:47.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.558+0000 7f214d447700 1 -- 172.21.15.88:0/2475980124 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2140009ae0 con 0x7f214814d7c0 2023-10-18T16:07:47.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.558+0000 7f214d447700 1 --2- 172.21.15.88:0/2475980124 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f214814d7c0 0x7f21480ab250 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f213800c470 tx=0x7f2138007c30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:47.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.558+0000 7f213dffb700 1 -- 172.21.15.88:0/2475980124 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21380072e0 con 0x7f214814d7c0 2023-10-18T16:07:47.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.558+0000 7f213dffb700 1 -- 172.21.15.88:0/2475980124 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2138007440 con 0x7f214814d7c0 2023-10-18T16:07:47.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.558+0000 7f214ec4a700 1 -- 172.21.15.88:0/2475980124 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f214814fef0 con 0x7f214814d7c0 2023-10-18T16:07:47.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.558+0000 7f213dffb700 1 -- 172.21.15.88:0/2475980124 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21380058f0 con 0x7f214814d7c0 2023-10-18T16:07:47.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.559+0000 7f214ec4a700 1 -- 172.21.15.88:0/2475980124 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2148160c70 con 0x7f214814d7c0 2023-10-18T16:07:47.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.560+0000 7f214ec4a700 1 -- 172.21.15.88:0/2475980124 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2148150080 con 0x7f214814d7c0 2023-10-18T16:07:47.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.560+0000 7f213dffb700 1 -- 172.21.15.88:0/2475980124 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f2138005ca0 con 0x7f214814d7c0 2023-10-18T16:07:47.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.560+0000 7f213dffb700 1 --2- 172.21.15.88:0/2475980124 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f212c05f150 0x7f212c061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:47.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.560+0000 7f213dffb700 1 -- 172.21.15.88:0/2475980124 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6121+0+0 (secure 0 0 0) 0x7f2138079840 con 0x7f214814d7c0 2023-10-18T16:07:47.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.560+0000 7f214dc48700 1 --2- 172.21.15.88:0/2475980124 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f212c05f150 0x7f212c061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:47.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.561+0000 7f214dc48700 1 --2- 172.21.15.88:0/2475980124 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f212c05f150 0x7f212c061610 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f214400bbc0 tx=0x7f2144008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:47.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.564+0000 7f213dffb700 1 -- 172.21.15.88:0/2475980124 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f213804ab10 con 0x7f214814d7c0 2023-10-18T16:07:47.728 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.724+0000 7f214ec4a700 1 -- 172.21.15.88:0/2475980124 --> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2148160c70 con 0x7f212c05f150 2023-10-18T16:07:47.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.733+0000 7f213dffb700 1 -- 172.21.15.88:0/2475980124 <== mgr.4102 v2:172.21.15.151:6832/90916 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+336380 (secure 0 0 0) 0x7f2148160c70 con 0x7f212c05f150 2023-10-18T16:07:47.741 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:07:47.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.737+0000 7f214ec4a700 1 -- 172.21.15.88:0/2475980124 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f212c05f150 msgr2=0x7f212c061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:47.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.737+0000 7f214ec4a700 1 --2- 172.21.15.88:0/2475980124 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f212c05f150 0x7f212c061610 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f214400bbc0 tx=0x7f2144008040).stop 2023-10-18T16:07:47.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.737+0000 7f214ec4a700 1 -- 172.21.15.88:0/2475980124 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f214814d7c0 msgr2=0x7f21480ab250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:47.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.737+0000 7f214ec4a700 1 --2- 172.21.15.88:0/2475980124 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f214814d7c0 0x7f21480ab250 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f213800c470 tx=0x7f2138007c30).stop 2023-10-18T16:07:47.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.737+0000 7f214ec4a700 1 -- 172.21.15.88:0/2475980124 shutdown_connections 2023-10-18T16:07:47.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.737+0000 7f214ec4a700 1 --2- 172.21.15.88:0/2475980124 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f212c05f150 0x7f212c061610 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:47.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.737+0000 7f214ec4a700 1 --2- 172.21.15.88:0/2475980124 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f214814dba0 0x7f21480ab740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:47.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.737+0000 7f214ec4a700 1 --2- 172.21.15.88:0/2475980124 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2148064360 0x7f2148150d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:47.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.737+0000 7f214ec4a700 1 --2- 172.21.15.88:0/2475980124 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f214814d7c0 0x7f21480ab250 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:47.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.737+0000 7f214ec4a700 1 -- 172.21.15.88:0/2475980124 >> 172.21.15.88:0/2475980124 conn(0x7f2148069d80 msgr2=0x7f2148154340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:47.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.737+0000 7f214ec4a700 1 -- 172.21.15.88:0/2475980124 shutdown_connections 2023-10-18T16:07:47.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.737+0000 7f214ec4a700 1 -- 172.21.15.88:0/2475980124 wait complete. 2023-10-18T16:07:47.746 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2023-10-18T16:07:47.752 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2023-10-18T16:07:46.457808+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":274,"acting":274,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":98,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2378500,"kb_used_data":1732,"kb_used_omap":0,"kb_used_meta":2376704,"kb_avail":752596220,"statfs":{"total":773094113280,"available":770658529280,"internally_reserved":0,"allocated":1773568,"data_stored":649208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2433744896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"4.1d","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.800856+0000","last_change":"2023-10-18T16:07:41.796671+0000","last_active":"2023-10-18T16:07:41.800856+0000","last_peered":"2023-10-18T16:07:41.800856+0000","last_clean":"2023-10-18T16:07:41.800856+0000","last_became_active":"2023-10-18T16:07:41.796451+0000","last_became_peered":"2023-10-18T16:07:41.796451+0000","last_unstale":"2023-10-18T16:07:41.800856+0000","last_undegraded":"2023-10-18T16:07:41.800856+0000","last_fullsized":"2023-10-18T16:07:41.800856+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.794828+0000","last_change":"2023-10-18T16:07:41.442111+0000","last_active":"2023-10-18T16:07:41.794828+0000","last_peered":"2023-10-18T16:07:41.794828+0000","last_clean":"2023-10-18T16:07:41.794828+0000","last_became_active":"2023-10-18T16:07:41.441815+0000","last_became_peered":"2023-10-18T16:07:41.441815+0000","last_unstale":"2023-10-18T16:07:41.794828+0000","last_undegraded":"2023-10-18T16:07:41.794828+0000","last_fullsized":"2023-10-18T16:07:41.794828+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":8,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.808517+0000","last_change":"2023-10-18T16:07:41.808517+0000","last_active":"2023-10-18T16:07:41.808517+0000","last_peered":"2023-10-18T16:07:41.808517+0000","last_clean":"2023-10-18T16:07:41.808517+0000","last_became_active":"2023-10-18T16:07:41.808224+0000","last_became_peered":"2023-10-18T16:07:41.808224+0000","last_unstale":"2023-10-18T16:07:41.808517+0000","last_undegraded":"2023-10-18T16:07:41.808517+0000","last_fullsized":"2023-10-18T16:07:41.808517+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.433589+0000","last_change":"2023-10-18T16:07:41.430021+0000","last_active":"2023-10-18T16:07:41.433589+0000","last_peered":"2023-10-18T16:07:41.433589+0000","last_clean":"2023-10-18T16:07:41.433589+0000","last_became_active":"2023-10-18T16:07:41.429781+0000","last_became_peered":"2023-10-18T16:07:41.429781+0000","last_unstale":"2023-10-18T16:07:41.433589+0000","last_undegraded":"2023-10-18T16:07:41.433589+0000","last_fullsized":"2023-10-18T16:07:41.433589+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":8,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.807746+0000","last_change":"2023-10-18T16:07:41.807746+0000","last_active":"2023-10-18T16:07:41.807746+0000","last_peered":"2023-10-18T16:07:41.807746+0000","last_clean":"2023-10-18T16:07:41.807746+0000","last_became_active":"2023-10-18T16:07:41.807256+0000","last_became_peered":"2023-10-18T16:07:41.807256+0000","last_unstale":"2023-10-18T16:07:41.807746+0000","last_undegraded":"2023-10-18T16:07:41.807746+0000","last_fullsized":"2023-10-18T16:07:41.807746+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.788944+0000","last_change":"2023-10-18T16:07:41.431122+0000","last_active":"2023-10-18T16:07:41.788944+0000","last_peered":"2023-10-18T16:07:41.788944+0000","last_clean":"2023-10-18T16:07:41.788944+0000","last_became_active":"2023-10-18T16:07:41.430831+0000","last_became_peered":"2023-10-18T16:07:41.430831+0000","last_unstale":"2023-10-18T16:07:41.788944+0000","last_undegraded":"2023-10-18T16:07:41.788944+0000","last_fullsized":"2023-10-18T16:07:41.788944+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.807269+0000","last_change":"2023-10-18T16:07:41.801866+0000","last_active":"2023-10-18T16:07:41.807269+0000","last_peered":"2023-10-18T16:07:41.807269+0000","last_clean":"2023-10-18T16:07:41.807269+0000","last_became_active":"2023-10-18T16:07:41.801602+0000","last_became_peered":"2023-10-18T16:07:41.801602+0000","last_unstale":"2023-10-18T16:07:41.807269+0000","last_undegraded":"2023-10-18T16:07:41.807269+0000","last_fullsized":"2023-10-18T16:07:41.807269+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.443089+0000","last_change":"2023-10-18T16:07:41.442448+0000","last_active":"2023-10-18T16:07:41.443089+0000","last_peered":"2023-10-18T16:07:41.443089+0000","last_clean":"2023-10-18T16:07:41.443089+0000","last_became_active":"2023-10-18T16:07:41.442143+0000","last_became_peered":"2023-10-18T16:07:41.442143+0000","last_unstale":"2023-10-18T16:07:41.443089+0000","last_undegraded":"2023-10-18T16:07:41.443089+0000","last_fullsized":"2023-10-18T16:07:41.443089+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.808841+0000","last_change":"2023-10-18T16:07:41.808302+0000","last_active":"2023-10-18T16:07:41.808841+0000","last_peered":"2023-10-18T16:07:41.808841+0000","last_clean":"2023-10-18T16:07:41.808841+0000","last_became_active":"2023-10-18T16:07:41.808001+0000","last_became_peered":"2023-10-18T16:07:41.808001+0000","last_unstale":"2023-10-18T16:07:41.808841+0000","last_undegraded":"2023-10-18T16:07:41.808841+0000","last_fullsized":"2023-10-18T16:07:41.808841+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.441462+0000","last_change":"2023-10-18T16:07:41.433080+0000","last_active":"2023-10-18T16:07:41.441462+0000","last_peered":"2023-10-18T16:07:41.441462+0000","last_clean":"2023-10-18T16:07:41.441462+0000","last_became_active":"2023-10-18T16:07:41.432714+0000","last_became_peered":"2023-10-18T16:07:41.432714+0000","last_unstale":"2023-10-18T16:07:41.441462+0000","last_undegraded":"2023-10-18T16:07:41.441462+0000","last_fullsized":"2023-10-18T16:07:41.441462+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":8,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.797599+0000","last_change":"2023-10-18T16:07:41.797599+0000","last_active":"2023-10-18T16:07:41.797599+0000","last_peered":"2023-10-18T16:07:41.797599+0000","last_clean":"2023-10-18T16:07:41.797599+0000","last_became_active":"2023-10-18T16:07:41.797318+0000","last_became_peered":"2023-10-18T16:07:41.797318+0000","last_unstale":"2023-10-18T16:07:41.797599+0000","last_undegraded":"2023-10-18T16:07:41.797599+0000","last_fullsized":"2023-10-18T16:07:41.797599+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.798325+0000","last_change":"2023-10-18T16:07:41.437425+0000","last_active":"2023-10-18T16:07:41.798325+0000","last_peered":"2023-10-18T16:07:41.798325+0000","last_clean":"2023-10-18T16:07:41.798325+0000","last_became_active":"2023-10-18T16:07:41.437135+0000","last_became_peered":"2023-10-18T16:07:41.437135+0000","last_unstale":"2023-10-18T16:07:41.798325+0000","last_undegraded":"2023-10-18T16:07:41.798325+0000","last_fullsized":"2023-10-18T16:07:41.798325+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.806231+0000","last_change":"2023-10-18T16:07:41.802908+0000","last_active":"2023-10-18T16:07:41.806231+0000","last_peered":"2023-10-18T16:07:41.806231+0000","last_clean":"2023-10-18T16:07:41.806231+0000","last_became_active":"2023-10-18T16:07:41.802644+0000","last_became_peered":"2023-10-18T16:07:41.802644+0000","last_unstale":"2023-10-18T16:07:41.806231+0000","last_undegraded":"2023-10-18T16:07:41.806231+0000","last_fullsized":"2023-10-18T16:07:41.806231+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.796534+0000","last_change":"2023-10-18T16:07:41.446842+0000","last_active":"2023-10-18T16:07:41.796534+0000","last_peered":"2023-10-18T16:07:41.796534+0000","last_clean":"2023-10-18T16:07:41.796534+0000","last_became_active":"2023-10-18T16:07:41.446567+0000","last_became_peered":"2023-10-18T16:07:41.446567+0000","last_unstale":"2023-10-18T16:07:41.796534+0000","last_undegraded":"2023-10-18T16:07:41.796534+0000","last_fullsized":"2023-10-18T16:07:41.796534+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.802559+0000","last_change":"2023-10-18T16:07:41.798736+0000","last_active":"2023-10-18T16:07:41.802559+0000","last_peered":"2023-10-18T16:07:41.802559+0000","last_clean":"2023-10-18T16:07:41.802559+0000","last_became_active":"2023-10-18T16:07:41.798458+0000","last_became_peered":"2023-10-18T16:07:41.798458+0000","last_unstale":"2023-10-18T16:07:41.802559+0000","last_undegraded":"2023-10-18T16:07:41.802559+0000","last_fullsized":"2023-10-18T16:07:41.802559+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.796971+0000","last_change":"2023-10-18T16:07:41.443369+0000","last_active":"2023-10-18T16:07:41.796971+0000","last_peered":"2023-10-18T16:07:41.796971+0000","last_clean":"2023-10-18T16:07:41.796971+0000","last_became_active":"2023-10-18T16:07:41.443064+0000","last_became_peered":"2023-10-18T16:07:41.443064+0000","last_unstale":"2023-10-18T16:07:41.796971+0000","last_undegraded":"2023-10-18T16:07:41.796971+0000","last_fullsized":"2023-10-18T16:07:41.796971+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":8,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.808196+0000","last_change":"2023-10-18T16:07:41.808196+0000","last_active":"2023-10-18T16:07:41.808196+0000","last_peered":"2023-10-18T16:07:41.808196+0000","last_clean":"2023-10-18T16:07:41.808196+0000","last_became_active":"2023-10-18T16:07:41.807537+0000","last_became_peered":"2023-10-18T16:07:41.807537+0000","last_unstale":"2023-10-18T16:07:41.808196+0000","last_undegraded":"2023-10-18T16:07:41.808196+0000","last_fullsized":"2023-10-18T16:07:41.808196+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.435794+0000","last_change":"2023-10-18T16:07:41.433498+0000","last_active":"2023-10-18T16:07:41.435794+0000","last_peered":"2023-10-18T16:07:41.435794+0000","last_clean":"2023-10-18T16:07:41.435794+0000","last_became_active":"2023-10-18T16:07:41.433321+0000","last_became_peered":"2023-10-18T16:07:41.433321+0000","last_unstale":"2023-10-18T16:07:41.435794+0000","last_undegraded":"2023-10-18T16:07:41.435794+0000","last_fullsized":"2023-10-18T16:07:41.435794+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":8,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.797372+0000","last_change":"2023-10-18T16:07:41.797372+0000","last_active":"2023-10-18T16:07:41.797372+0000","last_peered":"2023-10-18T16:07:41.797372+0000","last_clean":"2023-10-18T16:07:41.797372+0000","last_became_active":"2023-10-18T16:07:41.797108+0000","last_became_peered":"2023-10-18T16:07:41.797108+0000","last_unstale":"2023-10-18T16:07:41.797372+0000","last_undegraded":"2023-10-18T16:07:41.797372+0000","last_fullsized":"2023-10-18T16:07:41.797372+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.799678+0000","last_change":"2023-10-18T16:07:41.445469+0000","last_active":"2023-10-18T16:07:41.799678+0000","last_peered":"2023-10-18T16:07:41.799678+0000","last_clean":"2023-10-18T16:07:41.799678+0000","last_became_active":"2023-10-18T16:07:41.445204+0000","last_became_peered":"2023-10-18T16:07:41.445204+0000","last_unstale":"2023-10-18T16:07:41.799678+0000","last_undegraded":"2023-10-18T16:07:41.799678+0000","last_fullsized":"2023-10-18T16:07:41.799678+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.808614+0000","last_change":"2023-10-18T16:07:41.808046+0000","last_active":"2023-10-18T16:07:41.808614+0000","last_peered":"2023-10-18T16:07:41.808614+0000","last_clean":"2023-10-18T16:07:41.808614+0000","last_became_active":"2023-10-18T16:07:41.807189+0000","last_became_peered":"2023-10-18T16:07:41.807189+0000","last_unstale":"2023-10-18T16:07:41.808614+0000","last_undegraded":"2023-10-18T16:07:41.808614+0000","last_fullsized":"2023-10-18T16:07:41.808614+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.444649+0000","last_change":"2023-10-18T16:07:41.434313+0000","last_active":"2023-10-18T16:07:41.444649+0000","last_peered":"2023-10-18T16:07:41.444649+0000","last_clean":"2023-10-18T16:07:41.444649+0000","last_became_active":"2023-10-18T16:07:41.434022+0000","last_became_peered":"2023-10-18T16:07:41.434022+0000","last_unstale":"2023-10-18T16:07:41.444649+0000","last_undegraded":"2023-10-18T16:07:41.444649+0000","last_fullsized":"2023-10-18T16:07:41.444649+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":8,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.800593+0000","last_change":"2023-10-18T16:07:41.800593+0000","last_active":"2023-10-18T16:07:41.800593+0000","last_peered":"2023-10-18T16:07:41.800593+0000","last_clean":"2023-10-18T16:07:41.800593+0000","last_became_active":"2023-10-18T16:07:41.800302+0000","last_became_peered":"2023-10-18T16:07:41.800302+0000","last_unstale":"2023-10-18T16:07:41.800593+0000","last_undegraded":"2023-10-18T16:07:41.800593+0000","last_fullsized":"2023-10-18T16:07:41.800593+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.798209+0000","last_change":"2023-10-18T16:07:41.442977+0000","last_active":"2023-10-18T16:07:41.798209+0000","last_peered":"2023-10-18T16:07:41.798209+0000","last_clean":"2023-10-18T16:07:41.798209+0000","last_became_active":"2023-10-18T16:07:41.442383+0000","last_became_peered":"2023-10-18T16:07:41.442383+0000","last_unstale":"2023-10-18T16:07:41.798209+0000","last_undegraded":"2023-10-18T16:07:41.798209+0000","last_fullsized":"2023-10-18T16:07:41.798209+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.810814+0000","last_change":"2023-10-18T16:07:41.808758+0000","last_active":"2023-10-18T16:07:41.810814+0000","last_peered":"2023-10-18T16:07:41.810814+0000","last_clean":"2023-10-18T16:07:41.810814+0000","last_became_active":"2023-10-18T16:07:41.808475+0000","last_became_peered":"2023-10-18T16:07:41.808475+0000","last_unstale":"2023-10-18T16:07:41.810814+0000","last_undegraded":"2023-10-18T16:07:41.810814+0000","last_fullsized":"2023-10-18T16:07:41.810814+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.798959+0000","last_change":"2023-10-18T16:07:41.442566+0000","last_active":"2023-10-18T16:07:41.798959+0000","last_peered":"2023-10-18T16:07:41.798959+0000","last_clean":"2023-10-18T16:07:41.798959+0000","last_became_active":"2023-10-18T16:07:41.442264+0000","last_became_peered":"2023-10-18T16:07:41.442264+0000","last_unstale":"2023-10-18T16:07:41.798959+0000","last_undegraded":"2023-10-18T16:07:41.798959+0000","last_fullsized":"2023-10-18T16:07:41.798959+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":8,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.803408+0000","last_change":"2023-10-18T16:07:41.803408+0000","last_active":"2023-10-18T16:07:41.803408+0000","last_peered":"2023-10-18T16:07:41.803408+0000","last_clean":"2023-10-18T16:07:41.803408+0000","last_became_active":"2023-10-18T16:07:41.803142+0000","last_became_peered":"2023-10-18T16:07:41.803142+0000","last_unstale":"2023-10-18T16:07:41.803408+0000","last_undegraded":"2023-10-18T16:07:41.803408+0000","last_fullsized":"2023-10-18T16:07:41.803408+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.799082+0000","last_change":"2023-10-18T16:07:41.440460+0000","last_active":"2023-10-18T16:07:41.799082+0000","last_peered":"2023-10-18T16:07:41.799082+0000","last_clean":"2023-10-18T16:07:41.799082+0000","last_became_active":"2023-10-18T16:07:41.440162+0000","last_became_peered":"2023-10-18T16:07:41.440162+0000","last_unstale":"2023-10-18T16:07:41.799082+0000","last_undegraded":"2023-10-18T16:07:41.799082+0000","last_fullsized":"2023-10-18T16:07:41.799082+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":8,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.804730+0000","last_change":"2023-10-18T16:07:41.804730+0000","last_active":"2023-10-18T16:07:41.804730+0000","last_peered":"2023-10-18T16:07:41.804730+0000","last_clean":"2023-10-18T16:07:41.804730+0000","last_became_active":"2023-10-18T16:07:41.804455+0000","last_became_peered":"2023-10-18T16:07:41.804455+0000","last_unstale":"2023-10-18T16:07:41.804730+0000","last_undegraded":"2023-10-18T16:07:41.804730+0000","last_fullsized":"2023-10-18T16:07:41.804730+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.435773+0000","last_change":"2023-10-18T16:07:41.429156+0000","last_active":"2023-10-18T16:07:41.435773+0000","last_peered":"2023-10-18T16:07:41.435773+0000","last_clean":"2023-10-18T16:07:41.435773+0000","last_became_active":"2023-10-18T16:07:41.428828+0000","last_became_peered":"2023-10-18T16:07:41.428828+0000","last_unstale":"2023-10-18T16:07:41.435773+0000","last_undegraded":"2023-10-18T16:07:41.435773+0000","last_fullsized":"2023-10-18T16:07:41.435773+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.811080+0000","last_change":"2023-10-18T16:07:41.810261+0000","last_active":"2023-10-18T16:07:41.811080+0000","last_peered":"2023-10-18T16:07:41.811080+0000","last_clean":"2023-10-18T16:07:41.811080+0000","last_became_active":"2023-10-18T16:07:41.809938+0000","last_became_peered":"2023-10-18T16:07:41.809938+0000","last_unstale":"2023-10-18T16:07:41.811080+0000","last_undegraded":"2023-10-18T16:07:41.811080+0000","last_fullsized":"2023-10-18T16:07:41.811080+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.788799+0000","last_change":"2023-10-18T16:07:41.436527+0000","last_active":"2023-10-18T16:07:41.788799+0000","last_peered":"2023-10-18T16:07:41.788799+0000","last_clean":"2023-10-18T16:07:41.788799+0000","last_became_active":"2023-10-18T16:07:41.436223+0000","last_became_peered":"2023-10-18T16:07:41.436223+0000","last_unstale":"2023-10-18T16:07:41.788799+0000","last_undegraded":"2023-10-18T16:07:41.788799+0000","last_fullsized":"2023-10-18T16:07:41.788799+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.806186+0000","last_change":"2023-10-18T16:07:41.802220+0000","last_active":"2023-10-18T16:07:41.806186+0000","last_peered":"2023-10-18T16:07:41.806186+0000","last_clean":"2023-10-18T16:07:41.806186+0000","last_became_active":"2023-10-18T16:07:41.801909+0000","last_became_peered":"2023-10-18T16:07:41.801909+0000","last_unstale":"2023-10-18T16:07:41.806186+0000","last_undegraded":"2023-10-18T16:07:41.806186+0000","last_fullsized":"2023-10-18T16:07:41.806186+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.447846+0000","last_change":"2023-10-18T16:07:41.446075+0000","last_active":"2023-10-18T16:07:41.447846+0000","last_peered":"2023-10-18T16:07:41.447846+0000","last_clean":"2023-10-18T16:07:41.447846+0000","last_became_active":"2023-10-18T16:07:41.445817+0000","last_became_peered":"2023-10-18T16:07:41.445817+0000","last_unstale":"2023-10-18T16:07:41.447846+0000","last_undegraded":"2023-10-18T16:07:41.447846+0000","last_fullsized":"2023-10-18T16:07:41.447846+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.806073+0000","last_change":"2023-10-18T16:07:41.802289+0000","last_active":"2023-10-18T16:07:41.806073+0000","last_peered":"2023-10-18T16:07:41.806073+0000","last_clean":"2023-10-18T16:07:41.806073+0000","last_became_active":"2023-10-18T16:07:41.802038+0000","last_became_peered":"2023-10-18T16:07:41.802038+0000","last_unstale":"2023-10-18T16:07:41.806073+0000","last_undegraded":"2023-10-18T16:07:41.806073+0000","last_fullsized":"2023-10-18T16:07:41.806073+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.795284+0000","last_change":"2023-10-18T16:07:41.445650+0000","last_active":"2023-10-18T16:07:41.795284+0000","last_peered":"2023-10-18T16:07:41.795284+0000","last_clean":"2023-10-18T16:07:41.795284+0000","last_became_active":"2023-10-18T16:07:41.445297+0000","last_became_peered":"2023-10-18T16:07:41.445297+0000","last_unstale":"2023-10-18T16:07:41.795284+0000","last_undegraded":"2023-10-18T16:07:41.795284+0000","last_fullsized":"2023-10-18T16:07:41.795284+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":8,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.800350+0000","last_change":"2023-10-18T16:07:41.800350+0000","last_active":"2023-10-18T16:07:41.800350+0000","last_peered":"2023-10-18T16:07:41.800350+0000","last_clean":"2023-10-18T16:07:41.800350+0000","last_became_active":"2023-10-18T16:07:41.800095+0000","last_became_peered":"2023-10-18T16:07:41.800095+0000","last_unstale":"2023-10-18T16:07:41.800350+0000","last_undegraded":"2023-10-18T16:07:41.800350+0000","last_fullsized":"2023-10-18T16:07:41.800350+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.789807+0000","last_change":"2023-10-18T16:07:41.429381+0000","last_active":"2023-10-18T16:07:41.789807+0000","last_peered":"2023-10-18T16:07:41.789807+0000","last_clean":"2023-10-18T16:07:41.789807+0000","last_became_active":"2023-10-18T16:07:41.429117+0000","last_became_peered":"2023-10-18T16:07:41.429117+0000","last_unstale":"2023-10-18T16:07:41.789807+0000","last_undegraded":"2023-10-18T16:07:41.789807+0000","last_fullsized":"2023-10-18T16:07:41.789807+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.417130+0000","last_change":"2023-10-18T16:07:37.254194+0000","last_active":"2023-10-18T16:07:41.417130+0000","last_peered":"2023-10-18T16:07:41.417130+0000","last_clean":"2023-10-18T16:07:41.417130+0000","last_became_active":"2023-10-18T16:07:37.253900+0000","last_became_peered":"2023-10-18T16:07:37.253900+0000","last_unstale":"2023-10-18T16:07:41.417130+0000","last_undegraded":"2023-10-18T16:07:41.417130+0000","last_fullsized":"2023-10-18T16:07:41.417130+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:35.522314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:35.522314+0000","last_clean_scrub_stamp":"2023-10-18T16:07:35.522314+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.441430+0000","last_change":"2023-10-18T16:07:41.436498+0000","last_active":"2023-10-18T16:07:41.441430+0000","last_peered":"2023-10-18T16:07:41.441430+0000","last_clean":"2023-10-18T16:07:41.441430+0000","last_became_active":"2023-10-18T16:07:41.436182+0000","last_became_peered":"2023-10-18T16:07:41.436182+0000","last_unstale":"2023-10-18T16:07:41.441430+0000","last_undegraded":"2023-10-18T16:07:41.441430+0000","last_fullsized":"2023-10-18T16:07:41.441430+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":8,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.793823+0000","last_change":"2023-10-18T16:07:41.793823+0000","last_active":"2023-10-18T16:07:41.793823+0000","last_peered":"2023-10-18T16:07:41.793823+0000","last_clean":"2023-10-18T16:07:41.793823+0000","last_became_active":"2023-10-18T16:07:41.792600+0000","last_became_peered":"2023-10-18T16:07:41.792600+0000","last_unstale":"2023-10-18T16:07:41.793823+0000","last_undegraded":"2023-10-18T16:07:41.793823+0000","last_fullsized":"2023-10-18T16:07:41.793823+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.424416+0000","last_change":"2023-10-18T16:07:36.544503+0000","last_active":"2023-10-18T16:07:41.424416+0000","last_peered":"2023-10-18T16:07:41.424416+0000","last_clean":"2023-10-18T16:07:41.424416+0000","last_became_active":"2023-10-18T16:07:36.544254+0000","last_became_peered":"2023-10-18T16:07:36.544254+0000","last_unstale":"2023-10-18T16:07:41.424416+0000","last_undegraded":"2023-10-18T16:07:41.424416+0000","last_fullsized":"2023-10-18T16:07:41.424416+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:35.522314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:35.522314+0000","last_clean_scrub_stamp":"2023-10-18T16:07:35.522314+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.795162+0000","last_change":"2023-10-18T16:07:41.437532+0000","last_active":"2023-10-18T16:07:41.795162+0000","last_peered":"2023-10-18T16:07:41.795162+0000","last_clean":"2023-10-18T16:07:41.795162+0000","last_became_active":"2023-10-18T16:07:41.437257+0000","last_became_peered":"2023-10-18T16:07:41.437257+0000","last_unstale":"2023-10-18T16:07:41.795162+0000","last_undegraded":"2023-10-18T16:07:41.795162+0000","last_fullsized":"2023-10-18T16:07:41.795162+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":8,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.801719+0000","last_change":"2023-10-18T16:07:41.801719+0000","last_active":"2023-10-18T16:07:41.801719+0000","last_peered":"2023-10-18T16:07:41.801719+0000","last_clean":"2023-10-18T16:07:41.801719+0000","last_became_active":"2023-10-18T16:07:41.801325+0000","last_became_peered":"2023-10-18T16:07:41.801325+0000","last_unstale":"2023-10-18T16:07:41.801719+0000","last_undegraded":"2023-10-18T16:07:41.801719+0000","last_fullsized":"2023-10-18T16:07:41.801719+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":8,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.791251+0000","last_change":"2023-10-18T16:07:41.791251+0000","last_active":"2023-10-18T16:07:41.791251+0000","last_peered":"2023-10-18T16:07:41.791251+0000","last_clean":"2023-10-18T16:07:41.791251+0000","last_became_active":"2023-10-18T16:07:41.790992+0000","last_became_peered":"2023-10-18T16:07:41.790992+0000","last_unstale":"2023-10-18T16:07:41.791251+0000","last_undegraded":"2023-10-18T16:07:41.791251+0000","last_fullsized":"2023-10-18T16:07:41.791251+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.795059+0000","last_change":"2023-10-18T16:07:36.542401+0000","last_active":"2023-10-18T16:07:41.795059+0000","last_peered":"2023-10-18T16:07:41.795059+0000","last_clean":"2023-10-18T16:07:41.795059+0000","last_became_active":"2023-10-18T16:07:36.542071+0000","last_became_peered":"2023-10-18T16:07:36.542071+0000","last_unstale":"2023-10-18T16:07:41.795059+0000","last_undegraded":"2023-10-18T16:07:41.795059+0000","last_fullsized":"2023-10-18T16:07:41.795059+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:35.522314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:35.522314+0000","last_clean_scrub_stamp":"2023-10-18T16:07:35.522314+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.433392+0000","last_change":"2023-10-18T16:07:41.428730+0000","last_active":"2023-10-18T16:07:41.433392+0000","last_peered":"2023-10-18T16:07:41.433392+0000","last_clean":"2023-10-18T16:07:41.433392+0000","last_became_active":"2023-10-18T16:07:41.428469+0000","last_became_peered":"2023-10-18T16:07:41.428469+0000","last_unstale":"2023-10-18T16:07:41.433392+0000","last_undegraded":"2023-10-18T16:07:41.433392+0000","last_fullsized":"2023-10-18T16:07:41.433392+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":8,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.798241+0000","last_change":"2023-10-18T16:07:41.798241+0000","last_active":"2023-10-18T16:07:41.798241+0000","last_peered":"2023-10-18T16:07:41.798241+0000","last_clean":"2023-10-18T16:07:41.798241+0000","last_became_active":"2023-10-18T16:07:41.797958+0000","last_became_peered":"2023-10-18T16:07:41.797958+0000","last_unstale":"2023-10-18T16:07:41.798241+0000","last_undegraded":"2023-10-18T16:07:41.798241+0000","last_fullsized":"2023-10-18T16:07:41.798241+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.422240+0000","last_change":"2023-10-18T16:07:36.542651+0000","last_active":"2023-10-18T16:07:41.422240+0000","last_peered":"2023-10-18T16:07:41.422240+0000","last_clean":"2023-10-18T16:07:41.422240+0000","last_became_active":"2023-10-18T16:07:36.542306+0000","last_became_peered":"2023-10-18T16:07:36.542306+0000","last_unstale":"2023-10-18T16:07:41.422240+0000","last_undegraded":"2023-10-18T16:07:41.422240+0000","last_fullsized":"2023-10-18T16:07:41.422240+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:35.522314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:35.522314+0000","last_clean_scrub_stamp":"2023-10-18T16:07:35.522314+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.795394+0000","last_change":"2023-10-18T16:07:34.567589+0000","last_active":"2023-10-18T16:07:41.795394+0000","last_peered":"2023-10-18T16:07:41.795394+0000","last_clean":"2023-10-18T16:07:41.795394+0000","last_became_active":"2023-10-18T16:07:34.567301+0000","last_became_peered":"2023-10-18T16:07:34.567301+0000","last_unstale":"2023-10-18T16:07:41.795394+0000","last_undegraded":"2023-10-18T16:07:41.795394+0000","last_fullsized":"2023-10-18T16:07:41.795394+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:33.506699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:33.506699+0000","last_clean_scrub_stamp":"2023-10-18T16:07:33.506699+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.788528+0000","last_change":"2023-10-18T16:07:41.438974+0000","last_active":"2023-10-18T16:07:41.788528+0000","last_peered":"2023-10-18T16:07:41.788528+0000","last_clean":"2023-10-18T16:07:41.788528+0000","last_became_active":"2023-10-18T16:07:41.438710+0000","last_became_peered":"2023-10-18T16:07:41.438710+0000","last_unstale":"2023-10-18T16:07:41.788528+0000","last_undegraded":"2023-10-18T16:07:41.788528+0000","last_fullsized":"2023-10-18T16:07:41.788528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":8,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.800880+0000","last_change":"2023-10-18T16:07:41.800880+0000","last_active":"2023-10-18T16:07:41.800880+0000","last_peered":"2023-10-18T16:07:41.800880+0000","last_clean":"2023-10-18T16:07:41.800880+0000","last_became_active":"2023-10-18T16:07:41.800478+0000","last_became_peered":"2023-10-18T16:07:41.800478+0000","last_unstale":"2023-10-18T16:07:41.800880+0000","last_undegraded":"2023-10-18T16:07:41.800880+0000","last_fullsized":"2023-10-18T16:07:41.800880+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.795831+0000","last_change":"2023-10-18T16:07:36.542454+0000","last_active":"2023-10-18T16:07:41.795831+0000","last_peered":"2023-10-18T16:07:41.795831+0000","last_clean":"2023-10-18T16:07:41.795831+0000","last_became_active":"2023-10-18T16:07:36.542134+0000","last_became_peered":"2023-10-18T16:07:36.542134+0000","last_unstale":"2023-10-18T16:07:41.795831+0000","last_undegraded":"2023-10-18T16:07:41.795831+0000","last_fullsized":"2023-10-18T16:07:41.795831+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:35.522314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:35.522314+0000","last_clean_scrub_stamp":"2023-10-18T16:07:35.522314+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.446922+0000","last_change":"2023-10-18T16:07:41.445653+0000","last_active":"2023-10-18T16:07:41.446922+0000","last_peered":"2023-10-18T16:07:41.446922+0000","last_clean":"2023-10-18T16:07:41.446922+0000","last_became_active":"2023-10-18T16:07:41.445399+0000","last_became_peered":"2023-10-18T16:07:41.445399+0000","last_unstale":"2023-10-18T16:07:41.446922+0000","last_undegraded":"2023-10-18T16:07:41.446922+0000","last_fullsized":"2023-10-18T16:07:41.446922+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.798965+0000","last_change":"2023-10-18T16:07:37.254490+0000","last_active":"2023-10-18T16:07:41.798965+0000","last_peered":"2023-10-18T16:07:41.798965+0000","last_clean":"2023-10-18T16:07:41.798965+0000","last_became_active":"2023-10-18T16:07:37.254182+0000","last_became_peered":"2023-10-18T16:07:37.254182+0000","last_unstale":"2023-10-18T16:07:41.798965+0000","last_undegraded":"2023-10-18T16:07:41.798965+0000","last_fullsized":"2023-10-18T16:07:41.798965+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:35.522314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:35.522314+0000","last_clean_scrub_stamp":"2023-10-18T16:07:35.522314+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.433424+0000","last_change":"2023-10-18T16:07:41.428721+0000","last_active":"2023-10-18T16:07:41.433424+0000","last_peered":"2023-10-18T16:07:41.433424+0000","last_clean":"2023-10-18T16:07:41.433424+0000","last_became_active":"2023-10-18T16:07:41.428460+0000","last_became_peered":"2023-10-18T16:07:41.428460+0000","last_unstale":"2023-10-18T16:07:41.433424+0000","last_undegraded":"2023-10-18T16:07:41.433424+0000","last_fullsized":"2023-10-18T16:07:41.433424+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":8,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.801499+0000","last_change":"2023-10-18T16:07:41.801499+0000","last_active":"2023-10-18T16:07:41.801499+0000","last_peered":"2023-10-18T16:07:41.801499+0000","last_clean":"2023-10-18T16:07:41.801499+0000","last_became_active":"2023-10-18T16:07:41.800625+0000","last_became_peered":"2023-10-18T16:07:41.800625+0000","last_unstale":"2023-10-18T16:07:41.801499+0000","last_undegraded":"2023-10-18T16:07:41.801499+0000","last_fullsized":"2023-10-18T16:07:41.801499+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.789569+0000","last_change":"2023-10-18T16:07:41.435428+0000","last_active":"2023-10-18T16:07:41.789569+0000","last_peered":"2023-10-18T16:07:41.789569+0000","last_clean":"2023-10-18T16:07:41.789569+0000","last_became_active":"2023-10-18T16:07:41.435124+0000","last_became_peered":"2023-10-18T16:07:41.435124+0000","last_unstale":"2023-10-18T16:07:41.789569+0000","last_undegraded":"2023-10-18T16:07:41.789569+0000","last_fullsized":"2023-10-18T16:07:41.789569+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.416697+0000","last_change":"2023-10-18T16:07:36.542463+0000","last_active":"2023-10-18T16:07:41.416697+0000","last_peered":"2023-10-18T16:07:41.416697+0000","last_clean":"2023-10-18T16:07:41.416697+0000","last_became_active":"2023-10-18T16:07:36.542090+0000","last_became_peered":"2023-10-18T16:07:36.542090+0000","last_unstale":"2023-10-18T16:07:41.416697+0000","last_undegraded":"2023-10-18T16:07:41.416697+0000","last_fullsized":"2023-10-18T16:07:41.416697+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:35.522314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:35.522314+0000","last_clean_scrub_stamp":"2023-10-18T16:07:35.522314+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.807288+0000","last_change":"2023-10-18T16:07:41.802294+0000","last_active":"2023-10-18T16:07:41.807288+0000","last_peered":"2023-10-18T16:07:41.807288+0000","last_clean":"2023-10-18T16:07:41.807288+0000","last_became_active":"2023-10-18T16:07:41.801983+0000","last_became_peered":"2023-10-18T16:07:41.801983+0000","last_unstale":"2023-10-18T16:07:41.807288+0000","last_undegraded":"2023-10-18T16:07:41.807288+0000","last_fullsized":"2023-10-18T16:07:41.807288+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.428652+0000","last_change":"2023-10-18T16:07:36.540468+0000","last_active":"2023-10-18T16:07:41.428652+0000","last_peered":"2023-10-18T16:07:41.428652+0000","last_clean":"2023-10-18T16:07:41.428652+0000","last_became_active":"2023-10-18T16:07:36.540145+0000","last_became_peered":"2023-10-18T16:07:36.540145+0000","last_unstale":"2023-10-18T16:07:41.428652+0000","last_undegraded":"2023-10-18T16:07:41.428652+0000","last_fullsized":"2023-10-18T16:07:41.428652+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:35.522314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:35.522314+0000","last_clean_scrub_stamp":"2023-10-18T16:07:35.522314+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.435658+0000","last_change":"2023-10-18T16:07:41.431117+0000","last_active":"2023-10-18T16:07:41.435658+0000","last_peered":"2023-10-18T16:07:41.435658+0000","last_clean":"2023-10-18T16:07:41.435658+0000","last_became_active":"2023-10-18T16:07:41.430838+0000","last_became_peered":"2023-10-18T16:07:41.430838+0000","last_unstale":"2023-10-18T16:07:41.435658+0000","last_undegraded":"2023-10-18T16:07:41.435658+0000","last_fullsized":"2023-10-18T16:07:41.435658+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.806469+0000","last_change":"2023-10-18T16:07:41.803250+0000","last_active":"2023-10-18T16:07:41.806469+0000","last_peered":"2023-10-18T16:07:41.806469+0000","last_clean":"2023-10-18T16:07:41.806469+0000","last_became_active":"2023-10-18T16:07:41.802930+0000","last_became_peered":"2023-10-18T16:07:41.802930+0000","last_unstale":"2023-10-18T16:07:41.806469+0000","last_undegraded":"2023-10-18T16:07:41.806469+0000","last_fullsized":"2023-10-18T16:07:41.806469+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.807328+0000","last_change":"2023-10-18T16:07:41.803179+0000","last_active":"2023-10-18T16:07:41.807328+0000","last_peered":"2023-10-18T16:07:41.807328+0000","last_clean":"2023-10-18T16:07:41.807328+0000","last_became_active":"2023-10-18T16:07:41.802934+0000","last_became_peered":"2023-10-18T16:07:41.802934+0000","last_unstale":"2023-10-18T16:07:41.807328+0000","last_undegraded":"2023-10-18T16:07:41.807328+0000","last_fullsized":"2023-10-18T16:07:41.807328+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.444620+0000","last_change":"2023-10-18T16:07:41.443157+0000","last_active":"2023-10-18T16:07:41.444620+0000","last_peered":"2023-10-18T16:07:41.444620+0000","last_clean":"2023-10-18T16:07:41.444620+0000","last_became_active":"2023-10-18T16:07:41.442898+0000","last_became_peered":"2023-10-18T16:07:41.442898+0000","last_unstale":"2023-10-18T16:07:41.444620+0000","last_undegraded":"2023-10-18T16:07:41.444620+0000","last_fullsized":"2023-10-18T16:07:41.444620+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.807624+0000","last_change":"2023-10-18T16:07:41.804233+0000","last_active":"2023-10-18T16:07:41.807624+0000","last_peered":"2023-10-18T16:07:41.807624+0000","last_clean":"2023-10-18T16:07:41.807624+0000","last_became_active":"2023-10-18T16:07:41.803966+0000","last_became_peered":"2023-10-18T16:07:41.803966+0000","last_unstale":"2023-10-18T16:07:41.807624+0000","last_undegraded":"2023-10-18T16:07:41.807624+0000","last_fullsized":"2023-10-18T16:07:41.807624+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.447956+0000","last_change":"2023-10-18T16:07:41.443533+0000","last_active":"2023-10-18T16:07:41.447956+0000","last_peered":"2023-10-18T16:07:41.447956+0000","last_clean":"2023-10-18T16:07:41.447956+0000","last_became_active":"2023-10-18T16:07:41.443276+0000","last_became_peered":"2023-10-18T16:07:41.443276+0000","last_unstale":"2023-10-18T16:07:41.447956+0000","last_undegraded":"2023-10-18T16:07:41.447956+0000","last_fullsized":"2023-10-18T16:07:41.447956+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.805621+0000","last_change":"2023-10-18T16:07:41.799449+0000","last_active":"2023-10-18T16:07:41.805621+0000","last_peered":"2023-10-18T16:07:41.805621+0000","last_clean":"2023-10-18T16:07:41.805621+0000","last_became_active":"2023-10-18T16:07:41.799177+0000","last_became_peered":"2023-10-18T16:07:41.799177+0000","last_unstale":"2023-10-18T16:07:41.805621+0000","last_undegraded":"2023-10-18T16:07:41.805621+0000","last_fullsized":"2023-10-18T16:07:41.805621+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.448211+0000","last_change":"2023-10-18T16:07:41.443654+0000","last_active":"2023-10-18T16:07:41.448211+0000","last_peered":"2023-10-18T16:07:41.448211+0000","last_clean":"2023-10-18T16:07:41.448211+0000","last_became_active":"2023-10-18T16:07:41.443453+0000","last_became_peered":"2023-10-18T16:07:41.443453+0000","last_unstale":"2023-10-18T16:07:41.448211+0000","last_undegraded":"2023-10-18T16:07:41.448211+0000","last_fullsized":"2023-10-18T16:07:41.448211+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.802526+0000","last_change":"2023-10-18T16:07:41.798748+0000","last_active":"2023-10-18T16:07:41.802526+0000","last_peered":"2023-10-18T16:07:41.802526+0000","last_clean":"2023-10-18T16:07:41.802526+0000","last_became_active":"2023-10-18T16:07:41.798479+0000","last_became_peered":"2023-10-18T16:07:41.798479+0000","last_unstale":"2023-10-18T16:07:41.802526+0000","last_undegraded":"2023-10-18T16:07:41.802526+0000","last_fullsized":"2023-10-18T16:07:41.802526+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.788543+0000","last_change":"2023-10-18T16:07:41.438072+0000","last_active":"2023-10-18T16:07:41.788543+0000","last_peered":"2023-10-18T16:07:41.788543+0000","last_clean":"2023-10-18T16:07:41.788543+0000","last_became_active":"2023-10-18T16:07:41.437665+0000","last_became_peered":"2023-10-18T16:07:41.437665+0000","last_unstale":"2023-10-18T16:07:41.788543+0000","last_undegraded":"2023-10-18T16:07:41.788543+0000","last_fullsized":"2023-10-18T16:07:41.788543+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.806103+0000","last_change":"2023-10-18T16:07:41.800789+0000","last_active":"2023-10-18T16:07:41.806103+0000","last_peered":"2023-10-18T16:07:41.806103+0000","last_clean":"2023-10-18T16:07:41.806103+0000","last_became_active":"2023-10-18T16:07:41.800336+0000","last_became_peered":"2023-10-18T16:07:41.800336+0000","last_unstale":"2023-10-18T16:07:41.806103+0000","last_undegraded":"2023-10-18T16:07:41.806103+0000","last_fullsized":"2023-10-18T16:07:41.806103+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.440950+0000","last_change":"2023-10-18T16:07:41.439886+0000","last_active":"2023-10-18T16:07:41.440950+0000","last_peered":"2023-10-18T16:07:41.440950+0000","last_clean":"2023-10-18T16:07:41.440950+0000","last_became_active":"2023-10-18T16:07:41.439664+0000","last_became_peered":"2023-10-18T16:07:41.439664+0000","last_unstale":"2023-10-18T16:07:41.440950+0000","last_undegraded":"2023-10-18T16:07:41.440950+0000","last_fullsized":"2023-10-18T16:07:41.440950+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":8,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.797792+0000","last_change":"2023-10-18T16:07:41.797792+0000","last_active":"2023-10-18T16:07:41.797792+0000","last_peered":"2023-10-18T16:07:41.797792+0000","last_clean":"2023-10-18T16:07:41.797792+0000","last_became_active":"2023-10-18T16:07:41.797498+0000","last_became_peered":"2023-10-18T16:07:41.797498+0000","last_unstale":"2023-10-18T16:07:41.797792+0000","last_undegraded":"2023-10-18T16:07:41.797792+0000","last_fullsized":"2023-10-18T16:07:41.797792+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.439573+0000","last_change":"2023-10-18T16:07:41.434136+0000","last_active":"2023-10-18T16:07:41.439573+0000","last_peered":"2023-10-18T16:07:41.439573+0000","last_clean":"2023-10-18T16:07:41.439573+0000","last_became_active":"2023-10-18T16:07:41.433862+0000","last_became_peered":"2023-10-18T16:07:41.433862+0000","last_unstale":"2023-10-18T16:07:41.439573+0000","last_undegraded":"2023-10-18T16:07:41.439573+0000","last_fullsized":"2023-10-18T16:07:41.439573+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":8,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.797301+0000","last_change":"2023-10-18T16:07:41.797301+0000","last_active":"2023-10-18T16:07:41.797301+0000","last_peered":"2023-10-18T16:07:41.797301+0000","last_clean":"2023-10-18T16:07:41.797301+0000","last_became_active":"2023-10-18T16:07:41.797029+0000","last_became_peered":"2023-10-18T16:07:41.797029+0000","last_unstale":"2023-10-18T16:07:41.797301+0000","last_undegraded":"2023-10-18T16:07:41.797301+0000","last_fullsized":"2023-10-18T16:07:41.797301+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.795552+0000","last_change":"2023-10-18T16:07:41.426435+0000","last_active":"2023-10-18T16:07:41.795552+0000","last_peered":"2023-10-18T16:07:41.795552+0000","last_clean":"2023-10-18T16:07:41.795552+0000","last_became_active":"2023-10-18T16:07:41.426169+0000","last_became_peered":"2023-10-18T16:07:41.426169+0000","last_unstale":"2023-10-18T16:07:41.795552+0000","last_undegraded":"2023-10-18T16:07:41.795552+0000","last_fullsized":"2023-10-18T16:07:41.795552+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":8,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.808005+0000","last_change":"2023-10-18T16:07:41.808005+0000","last_active":"2023-10-18T16:07:41.808005+0000","last_peered":"2023-10-18T16:07:41.808005+0000","last_clean":"2023-10-18T16:07:41.808005+0000","last_became_active":"2023-10-18T16:07:41.807155+0000","last_became_peered":"2023-10-18T16:07:41.807155+0000","last_unstale":"2023-10-18T16:07:41.808005+0000","last_undegraded":"2023-10-18T16:07:41.808005+0000","last_fullsized":"2023-10-18T16:07:41.808005+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.446607+0000","last_change":"2023-10-18T16:07:41.437945+0000","last_active":"2023-10-18T16:07:41.446607+0000","last_peered":"2023-10-18T16:07:41.446607+0000","last_clean":"2023-10-18T16:07:41.446607+0000","last_became_active":"2023-10-18T16:07:41.437118+0000","last_became_peered":"2023-10-18T16:07:41.437118+0000","last_unstale":"2023-10-18T16:07:41.446607+0000","last_undegraded":"2023-10-18T16:07:41.446607+0000","last_fullsized":"2023-10-18T16:07:41.446607+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":8,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.801600+0000","last_change":"2023-10-18T16:07:41.801600+0000","last_active":"2023-10-18T16:07:41.801600+0000","last_peered":"2023-10-18T16:07:41.801600+0000","last_clean":"2023-10-18T16:07:41.801600+0000","last_became_active":"2023-10-18T16:07:41.801002+0000","last_became_peered":"2023-10-18T16:07:41.801002+0000","last_unstale":"2023-10-18T16:07:41.801600+0000","last_undegraded":"2023-10-18T16:07:41.801600+0000","last_fullsized":"2023-10-18T16:07:41.801600+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.435825+0000","last_change":"2023-10-18T16:07:41.432368+0000","last_active":"2023-10-18T16:07:41.435825+0000","last_peered":"2023-10-18T16:07:41.435825+0000","last_clean":"2023-10-18T16:07:41.435825+0000","last_became_active":"2023-10-18T16:07:41.431923+0000","last_became_peered":"2023-10-18T16:07:41.431923+0000","last_unstale":"2023-10-18T16:07:41.435825+0000","last_undegraded":"2023-10-18T16:07:41.435825+0000","last_fullsized":"2023-10-18T16:07:41.435825+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.810702+0000","last_change":"2023-10-18T16:07:41.809593+0000","last_active":"2023-10-18T16:07:41.810702+0000","last_peered":"2023-10-18T16:07:41.810702+0000","last_clean":"2023-10-18T16:07:41.810702+0000","last_became_active":"2023-10-18T16:07:41.809309+0000","last_became_peered":"2023-10-18T16:07:41.809309+0000","last_unstale":"2023-10-18T16:07:41.810702+0000","last_undegraded":"2023-10-18T16:07:41.810702+0000","last_fullsized":"2023-10-18T16:07:41.810702+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.789348+0000","last_change":"2023-10-18T16:07:41.443916+0000","last_active":"2023-10-18T16:07:41.789348+0000","last_peered":"2023-10-18T16:07:41.789348+0000","last_clean":"2023-10-18T16:07:41.789348+0000","last_became_active":"2023-10-18T16:07:41.443695+0000","last_became_peered":"2023-10-18T16:07:41.443695+0000","last_unstale":"2023-10-18T16:07:41.789348+0000","last_undegraded":"2023-10-18T16:07:41.789348+0000","last_fullsized":"2023-10-18T16:07:41.789348+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.806255+0000","last_change":"2023-10-18T16:07:41.803821+0000","last_active":"2023-10-18T16:07:41.806255+0000","last_peered":"2023-10-18T16:07:41.806255+0000","last_clean":"2023-10-18T16:07:41.806255+0000","last_became_active":"2023-10-18T16:07:41.803540+0000","last_became_peered":"2023-10-18T16:07:41.803540+0000","last_unstale":"2023-10-18T16:07:41.806255+0000","last_undegraded":"2023-10-18T16:07:41.806255+0000","last_fullsized":"2023-10-18T16:07:41.806255+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.433349+0000","last_change":"2023-10-18T16:07:41.429591+0000","last_active":"2023-10-18T16:07:41.433349+0000","last_peered":"2023-10-18T16:07:41.433349+0000","last_clean":"2023-10-18T16:07:41.433349+0000","last_became_active":"2023-10-18T16:07:41.429290+0000","last_became_peered":"2023-10-18T16:07:41.429290+0000","last_unstale":"2023-10-18T16:07:41.433349+0000","last_undegraded":"2023-10-18T16:07:41.433349+0000","last_fullsized":"2023-10-18T16:07:41.433349+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.799825+0000","last_change":"2023-10-18T16:07:41.795679+0000","last_active":"2023-10-18T16:07:41.799825+0000","last_peered":"2023-10-18T16:07:41.799825+0000","last_clean":"2023-10-18T16:07:41.799825+0000","last_became_active":"2023-10-18T16:07:41.795413+0000","last_became_peered":"2023-10-18T16:07:41.795413+0000","last_unstale":"2023-10-18T16:07:41.799825+0000","last_undegraded":"2023-10-18T16:07:41.799825+0000","last_fullsized":"2023-10-18T16:07:41.799825+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.444766+0000","last_change":"2023-10-18T16:07:41.436812+0000","last_active":"2023-10-18T16:07:41.444766+0000","last_peered":"2023-10-18T16:07:41.444766+0000","last_clean":"2023-10-18T16:07:41.444766+0000","last_became_active":"2023-10-18T16:07:41.436541+0000","last_became_peered":"2023-10-18T16:07:41.436541+0000","last_unstale":"2023-10-18T16:07:41.444766+0000","last_undegraded":"2023-10-18T16:07:41.444766+0000","last_fullsized":"2023-10-18T16:07:41.444766+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":8,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.803115+0000","last_change":"2023-10-18T16:07:41.803115+0000","last_active":"2023-10-18T16:07:41.803115+0000","last_peered":"2023-10-18T16:07:41.803115+0000","last_clean":"2023-10-18T16:07:41.803115+0000","last_became_active":"2023-10-18T16:07:41.802828+0000","last_became_peered":"2023-10-18T16:07:41.802828+0000","last_unstale":"2023-10-18T16:07:41.803115+0000","last_undegraded":"2023-10-18T16:07:41.803115+0000","last_fullsized":"2023-10-18T16:07:41.803115+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.789662+0000","last_change":"2023-10-18T16:07:41.443476+0000","last_active":"2023-10-18T16:07:41.789662+0000","last_peered":"2023-10-18T16:07:41.789662+0000","last_clean":"2023-10-18T16:07:41.789662+0000","last_became_active":"2023-10-18T16:07:41.443295+0000","last_became_peered":"2023-10-18T16:07:41.443295+0000","last_unstale":"2023-10-18T16:07:41.789662+0000","last_undegraded":"2023-10-18T16:07:41.789662+0000","last_fullsized":"2023-10-18T16:07:41.789662+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":8,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.800058+0000","last_change":"2023-10-18T16:07:41.800058+0000","last_active":"2023-10-18T16:07:41.800058+0000","last_peered":"2023-10-18T16:07:41.800058+0000","last_clean":"2023-10-18T16:07:41.800058+0000","last_became_active":"2023-10-18T16:07:41.799452+0000","last_became_peered":"2023-10-18T16:07:41.799452+0000","last_unstale":"2023-10-18T16:07:41.800058+0000","last_undegraded":"2023-10-18T16:07:41.800058+0000","last_fullsized":"2023-10-18T16:07:41.800058+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.446753+0000","last_change":"2023-10-18T16:07:41.440724+0000","last_active":"2023-10-18T16:07:41.446753+0000","last_peered":"2023-10-18T16:07:41.446753+0000","last_clean":"2023-10-18T16:07:41.446753+0000","last_became_active":"2023-10-18T16:07:41.440435+0000","last_became_peered":"2023-10-18T16:07:41.440435+0000","last_unstale":"2023-10-18T16:07:41.446753+0000","last_undegraded":"2023-10-18T16:07:41.446753+0000","last_fullsized":"2023-10-18T16:07:41.446753+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.806559+0000","last_change":"2023-10-18T16:07:41.804438+0000","last_active":"2023-10-18T16:07:41.806559+0000","last_peered":"2023-10-18T16:07:41.806559+0000","last_clean":"2023-10-18T16:07:41.806559+0000","last_became_active":"2023-10-18T16:07:41.804126+0000","last_became_peered":"2023-10-18T16:07:41.804126+0000","last_unstale":"2023-10-18T16:07:41.806559+0000","last_undegraded":"2023-10-18T16:07:41.806559+0000","last_fullsized":"2023-10-18T16:07:41.806559+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.788864+0000","last_change":"2023-10-18T16:07:41.447812+0000","last_active":"2023-10-18T16:07:41.788864+0000","last_peered":"2023-10-18T16:07:41.788864+0000","last_clean":"2023-10-18T16:07:41.788864+0000","last_became_active":"2023-10-18T16:07:41.447479+0000","last_became_peered":"2023-10-18T16:07:41.447479+0000","last_unstale":"2023-10-18T16:07:41.788864+0000","last_undegraded":"2023-10-18T16:07:41.788864+0000","last_fullsized":"2023-10-18T16:07:41.788864+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.804861+0000","last_change":"2023-10-18T16:07:41.801084+0000","last_active":"2023-10-18T16:07:41.804861+0000","last_peered":"2023-10-18T16:07:41.804861+0000","last_clean":"2023-10-18T16:07:41.804861+0000","last_became_active":"2023-10-18T16:07:41.800783+0000","last_became_peered":"2023-10-18T16:07:41.800783+0000","last_unstale":"2023-10-18T16:07:41.804861+0000","last_undegraded":"2023-10-18T16:07:41.804861+0000","last_fullsized":"2023-10-18T16:07:41.804861+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.446593+0000","last_change":"2023-10-18T16:07:41.437755+0000","last_active":"2023-10-18T16:07:41.446593+0000","last_peered":"2023-10-18T16:07:41.446593+0000","last_clean":"2023-10-18T16:07:41.446593+0000","last_became_active":"2023-10-18T16:07:41.436781+0000","last_became_peered":"2023-10-18T16:07:41.436781+0000","last_unstale":"2023-10-18T16:07:41.446593+0000","last_undegraded":"2023-10-18T16:07:41.446593+0000","last_fullsized":"2023-10-18T16:07:41.446593+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.810938+0000","last_change":"2023-10-18T16:07:41.808318+0000","last_active":"2023-10-18T16:07:41.810938+0000","last_peered":"2023-10-18T16:07:41.810938+0000","last_clean":"2023-10-18T16:07:41.810938+0000","last_became_active":"2023-10-18T16:07:41.807983+0000","last_became_peered":"2023-10-18T16:07:41.807983+0000","last_unstale":"2023-10-18T16:07:41.810938+0000","last_undegraded":"2023-10-18T16:07:41.810938+0000","last_fullsized":"2023-10-18T16:07:41.810938+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.443251+0000","last_change":"2023-10-18T16:07:41.442119+0000","last_active":"2023-10-18T16:07:41.443251+0000","last_peered":"2023-10-18T16:07:41.443251+0000","last_clean":"2023-10-18T16:07:41.443251+0000","last_became_active":"2023-10-18T16:07:41.441816+0000","last_became_peered":"2023-10-18T16:07:41.441816+0000","last_unstale":"2023-10-18T16:07:41.443251+0000","last_undegraded":"2023-10-18T16:07:41.443251+0000","last_fullsized":"2023-10-18T16:07:41.443251+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.810523+0000","last_change":"2023-10-18T16:07:41.809378+0000","last_active":"2023-10-18T16:07:41.810523+0000","last_peered":"2023-10-18T16:07:41.810523+0000","last_clean":"2023-10-18T16:07:41.810523+0000","last_became_active":"2023-10-18T16:07:41.809102+0000","last_became_peered":"2023-10-18T16:07:41.809102+0000","last_unstale":"2023-10-18T16:07:41.810523+0000","last_undegraded":"2023-10-18T16:07:41.810523+0000","last_fullsized":"2023-10-18T16:07:41.810523+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.447410+0000","last_change":"2023-10-18T16:07:41.442792+0000","last_active":"2023-10-18T16:07:41.447410+0000","last_peered":"2023-10-18T16:07:41.447410+0000","last_clean":"2023-10-18T16:07:41.447410+0000","last_became_active":"2023-10-18T16:07:41.442506+0000","last_became_peered":"2023-10-18T16:07:41.442506+0000","last_unstale":"2023-10-18T16:07:41.447410+0000","last_undegraded":"2023-10-18T16:07:41.447410+0000","last_fullsized":"2023-10-18T16:07:41.447410+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":8,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.804704+0000","last_change":"2023-10-18T16:07:41.804704+0000","last_active":"2023-10-18T16:07:41.804704+0000","last_peered":"2023-10-18T16:07:41.804704+0000","last_clean":"2023-10-18T16:07:41.804704+0000","last_became_active":"2023-10-18T16:07:41.804437+0000","last_became_peered":"2023-10-18T16:07:41.804437+0000","last_unstale":"2023-10-18T16:07:41.804704+0000","last_undegraded":"2023-10-18T16:07:41.804704+0000","last_fullsized":"2023-10-18T16:07:41.804704+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.798211+0000","last_change":"2023-10-18T16:07:41.436171+0000","last_active":"2023-10-18T16:07:41.798211+0000","last_peered":"2023-10-18T16:07:41.798211+0000","last_clean":"2023-10-18T16:07:41.798211+0000","last_became_active":"2023-10-18T16:07:41.435844+0000","last_became_peered":"2023-10-18T16:07:41.435844+0000","last_unstale":"2023-10-18T16:07:41.798211+0000","last_undegraded":"2023-10-18T16:07:41.798211+0000","last_fullsized":"2023-10-18T16:07:41.798211+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.802055+0000","last_change":"2023-10-18T16:07:41.796556+0000","last_active":"2023-10-18T16:07:41.802055+0000","last_peered":"2023-10-18T16:07:41.802055+0000","last_clean":"2023-10-18T16:07:41.802055+0000","last_became_active":"2023-10-18T16:07:41.796291+0000","last_became_peered":"2023-10-18T16:07:41.796291+0000","last_unstale":"2023-10-18T16:07:41.802055+0000","last_undegraded":"2023-10-18T16:07:41.802055+0000","last_fullsized":"2023-10-18T16:07:41.802055+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.795432+0000","last_change":"2023-10-18T16:07:41.434332+0000","last_active":"2023-10-18T16:07:41.795432+0000","last_peered":"2023-10-18T16:07:41.795432+0000","last_clean":"2023-10-18T16:07:41.795432+0000","last_became_active":"2023-10-18T16:07:41.434076+0000","last_became_peered":"2023-10-18T16:07:41.434076+0000","last_unstale":"2023-10-18T16:07:41.795432+0000","last_undegraded":"2023-10-18T16:07:41.795432+0000","last_fullsized":"2023-10-18T16:07:41.795432+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.808487+0000","last_change":"2023-10-18T16:07:41.806379+0000","last_active":"2023-10-18T16:07:41.808487+0000","last_peered":"2023-10-18T16:07:41.808487+0000","last_clean":"2023-10-18T16:07:41.808487+0000","last_became_active":"2023-10-18T16:07:41.806018+0000","last_became_peered":"2023-10-18T16:07:41.806018+0000","last_unstale":"2023-10-18T16:07:41.808487+0000","last_undegraded":"2023-10-18T16:07:41.808487+0000","last_fullsized":"2023-10-18T16:07:41.808487+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.446654+0000","last_change":"2023-10-18T16:07:41.438688+0000","last_active":"2023-10-18T16:07:41.446654+0000","last_peered":"2023-10-18T16:07:41.446654+0000","last_clean":"2023-10-18T16:07:41.446654+0000","last_became_active":"2023-10-18T16:07:41.438405+0000","last_became_peered":"2023-10-18T16:07:41.438405+0000","last_unstale":"2023-10-18T16:07:41.446654+0000","last_undegraded":"2023-10-18T16:07:41.446654+0000","last_fullsized":"2023-10-18T16:07:41.446654+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":8,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.800230+0000","last_change":"2023-10-18T16:07:41.800230+0000","last_active":"2023-10-18T16:07:41.800230+0000","last_peered":"2023-10-18T16:07:41.800230+0000","last_clean":"2023-10-18T16:07:41.800230+0000","last_became_active":"2023-10-18T16:07:41.799787+0000","last_became_peered":"2023-10-18T16:07:41.799787+0000","last_unstale":"2023-10-18T16:07:41.800230+0000","last_undegraded":"2023-10-18T16:07:41.800230+0000","last_fullsized":"2023-10-18T16:07:41.800230+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.446769+0000","last_change":"2023-10-18T16:07:41.440794+0000","last_active":"2023-10-18T16:07:41.446769+0000","last_peered":"2023-10-18T16:07:41.446769+0000","last_clean":"2023-10-18T16:07:41.446769+0000","last_became_active":"2023-10-18T16:07:41.440412+0000","last_became_peered":"2023-10-18T16:07:41.440412+0000","last_unstale":"2023-10-18T16:07:41.446769+0000","last_undegraded":"2023-10-18T16:07:41.446769+0000","last_fullsized":"2023-10-18T16:07:41.446769+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":8,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.801402+0000","last_change":"2023-10-18T16:07:41.801402+0000","last_active":"2023-10-18T16:07:41.801402+0000","last_peered":"2023-10-18T16:07:41.801402+0000","last_clean":"2023-10-18T16:07:41.801402+0000","last_became_active":"2023-10-18T16:07:41.800562+0000","last_became_peered":"2023-10-18T16:07:41.800562+0000","last_unstale":"2023-10-18T16:07:41.801402+0000","last_undegraded":"2023-10-18T16:07:41.801402+0000","last_fullsized":"2023-10-18T16:07:41.801402+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.446301+0000","last_change":"2023-10-18T16:07:41.445361+0000","last_active":"2023-10-18T16:07:41.446301+0000","last_peered":"2023-10-18T16:07:41.446301+0000","last_clean":"2023-10-18T16:07:41.446301+0000","last_became_active":"2023-10-18T16:07:41.445159+0000","last_became_peered":"2023-10-18T16:07:41.445159+0000","last_unstale":"2023-10-18T16:07:41.446301+0000","last_undegraded":"2023-10-18T16:07:41.446301+0000","last_fullsized":"2023-10-18T16:07:41.446301+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.806606+0000","last_change":"2023-10-18T16:07:41.806046+0000","last_active":"2023-10-18T16:07:41.806606+0000","last_peered":"2023-10-18T16:07:41.806606+0000","last_clean":"2023-10-18T16:07:41.806606+0000","last_became_active":"2023-10-18T16:07:41.805799+0000","last_became_peered":"2023-10-18T16:07:41.805799+0000","last_unstale":"2023-10-18T16:07:41.806606+0000","last_undegraded":"2023-10-18T16:07:41.806606+0000","last_fullsized":"2023-10-18T16:07:41.806606+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.430087+0000","last_change":"2023-10-18T16:07:41.425946+0000","last_active":"2023-10-18T16:07:41.430087+0000","last_peered":"2023-10-18T16:07:41.430087+0000","last_clean":"2023-10-18T16:07:41.430087+0000","last_became_active":"2023-10-18T16:07:41.425673+0000","last_became_peered":"2023-10-18T16:07:41.425673+0000","last_unstale":"2023-10-18T16:07:41.430087+0000","last_undegraded":"2023-10-18T16:07:41.430087+0000","last_fullsized":"2023-10-18T16:07:41.430087+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.811135+0000","last_change":"2023-10-18T16:07:41.810501+0000","last_active":"2023-10-18T16:07:41.811135+0000","last_peered":"2023-10-18T16:07:41.811135+0000","last_clean":"2023-10-18T16:07:41.811135+0000","last_became_active":"2023-10-18T16:07:41.810233+0000","last_became_peered":"2023-10-18T16:07:41.810233+0000","last_unstale":"2023-10-18T16:07:41.811135+0000","last_undegraded":"2023-10-18T16:07:41.811135+0000","last_fullsized":"2023-10-18T16:07:41.811135+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.788717+0000","last_change":"2023-10-18T16:07:41.434399+0000","last_active":"2023-10-18T16:07:41.788717+0000","last_peered":"2023-10-18T16:07:41.788717+0000","last_clean":"2023-10-18T16:07:41.788717+0000","last_became_active":"2023-10-18T16:07:41.434143+0000","last_became_peered":"2023-10-18T16:07:41.434143+0000","last_unstale":"2023-10-18T16:07:41.788717+0000","last_undegraded":"2023-10-18T16:07:41.788717+0000","last_fullsized":"2023-10-18T16:07:41.788717+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.801283+0000","last_change":"2023-10-18T16:07:41.799082+0000","last_active":"2023-10-18T16:07:41.801283+0000","last_peered":"2023-10-18T16:07:41.801283+0000","last_clean":"2023-10-18T16:07:41.801283+0000","last_became_active":"2023-10-18T16:07:41.798807+0000","last_became_peered":"2023-10-18T16:07:41.798807+0000","last_unstale":"2023-10-18T16:07:41.801283+0000","last_undegraded":"2023-10-18T16:07:41.801283+0000","last_fullsized":"2023-10-18T16:07:41.801283+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.796407+0000","last_change":"2023-10-18T16:07:41.445579+0000","last_active":"2023-10-18T16:07:41.796407+0000","last_peered":"2023-10-18T16:07:41.796407+0000","last_clean":"2023-10-18T16:07:41.796407+0000","last_became_active":"2023-10-18T16:07:41.445348+0000","last_became_peered":"2023-10-18T16:07:41.445348+0000","last_unstale":"2023-10-18T16:07:41.796407+0000","last_undegraded":"2023-10-18T16:07:41.796407+0000","last_fullsized":"2023-10-18T16:07:41.796407+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":8,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.802347+0000","last_change":"2023-10-18T16:07:41.802347+0000","last_active":"2023-10-18T16:07:41.802347+0000","last_peered":"2023-10-18T16:07:41.802347+0000","last_clean":"2023-10-18T16:07:41.802347+0000","last_became_active":"2023-10-18T16:07:41.802061+0000","last_became_peered":"2023-10-18T16:07:41.802061+0000","last_unstale":"2023-10-18T16:07:41.802347+0000","last_undegraded":"2023-10-18T16:07:41.802347+0000","last_fullsized":"2023-10-18T16:07:41.802347+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.447357+0000","last_change":"2023-10-18T16:07:41.439414+0000","last_active":"2023-10-18T16:07:41.447357+0000","last_peered":"2023-10-18T16:07:41.447357+0000","last_clean":"2023-10-18T16:07:41.447357+0000","last_became_active":"2023-10-18T16:07:41.439137+0000","last_became_peered":"2023-10-18T16:07:41.439137+0000","last_unstale":"2023-10-18T16:07:41.447357+0000","last_undegraded":"2023-10-18T16:07:41.447357+0000","last_fullsized":"2023-10-18T16:07:41.447357+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.810429+0000","last_change":"2023-10-18T16:07:41.808426+0000","last_active":"2023-10-18T16:07:41.810429+0000","last_peered":"2023-10-18T16:07:41.810429+0000","last_clean":"2023-10-18T16:07:41.810429+0000","last_became_active":"2023-10-18T16:07:41.808146+0000","last_became_peered":"2023-10-18T16:07:41.808146+0000","last_unstale":"2023-10-18T16:07:41.810429+0000","last_undegraded":"2023-10-18T16:07:41.810429+0000","last_fullsized":"2023-10-18T16:07:41.810429+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.447293+0000","last_change":"2023-10-18T16:07:41.439810+0000","last_active":"2023-10-18T16:07:41.447293+0000","last_peered":"2023-10-18T16:07:41.447293+0000","last_clean":"2023-10-18T16:07:41.447293+0000","last_became_active":"2023-10-18T16:07:41.439545+0000","last_became_peered":"2023-10-18T16:07:41.439545+0000","last_unstale":"2023-10-18T16:07:41.447293+0000","last_undegraded":"2023-10-18T16:07:41.447293+0000","last_fullsized":"2023-10-18T16:07:41.447293+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.806045+0000","last_change":"2023-10-18T16:07:41.800617+0000","last_active":"2023-10-18T16:07:41.806045+0000","last_peered":"2023-10-18T16:07:41.806045+0000","last_clean":"2023-10-18T16:07:41.806045+0000","last_became_active":"2023-10-18T16:07:41.799998+0000","last_became_peered":"2023-10-18T16:07:41.799998+0000","last_unstale":"2023-10-18T16:07:41.806045+0000","last_undegraded":"2023-10-18T16:07:41.806045+0000","last_fullsized":"2023-10-18T16:07:41.806045+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.798396+0000","last_change":"2023-10-18T16:07:41.442796+0000","last_active":"2023-10-18T16:07:41.798396+0000","last_peered":"2023-10-18T16:07:41.798396+0000","last_clean":"2023-10-18T16:07:41.798396+0000","last_became_active":"2023-10-18T16:07:41.442007+0000","last_became_peered":"2023-10-18T16:07:41.442007+0000","last_unstale":"2023-10-18T16:07:41.798396+0000","last_undegraded":"2023-10-18T16:07:41.798396+0000","last_fullsized":"2023-10-18T16:07:41.798396+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3f","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.801310+0000","last_change":"2023-10-18T16:07:41.799046+0000","last_active":"2023-10-18T16:07:41.801310+0000","last_peered":"2023-10-18T16:07:41.801310+0000","last_clean":"2023-10-18T16:07:41.801310+0000","last_became_active":"2023-10-18T16:07:41.798754+0000","last_became_peered":"2023-10-18T16:07:41.798754+0000","last_unstale":"2023-10-18T16:07:41.801310+0000","last_undegraded":"2023-10-18T16:07:41.801310+0000","last_fullsized":"2023-10-18T16:07:41.801310+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.789919+0000","last_change":"2023-10-18T16:07:41.441443+0000","last_active":"2023-10-18T16:07:41.789919+0000","last_peered":"2023-10-18T16:07:41.789919+0000","last_clean":"2023-10-18T16:07:41.789919+0000","last_became_active":"2023-10-18T16:07:41.441208+0000","last_became_peered":"2023-10-18T16:07:41.441208+0000","last_unstale":"2023-10-18T16:07:41.789919+0000","last_undegraded":"2023-10-18T16:07:41.789919+0000","last_fullsized":"2023-10-18T16:07:41.789919+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":8,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.800460+0000","last_change":"2023-10-18T16:07:41.800460+0000","last_active":"2023-10-18T16:07:41.800460+0000","last_peered":"2023-10-18T16:07:41.800460+0000","last_clean":"2023-10-18T16:07:41.800460+0000","last_became_active":"2023-10-18T16:07:41.800013+0000","last_became_peered":"2023-10-18T16:07:41.800013+0000","last_unstale":"2023-10-18T16:07:41.800460+0000","last_undegraded":"2023-10-18T16:07:41.800460+0000","last_fullsized":"2023-10-18T16:07:41.800460+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.799786+0000","last_change":"2023-10-18T16:07:41.438711+0000","last_active":"2023-10-18T16:07:41.799786+0000","last_peered":"2023-10-18T16:07:41.799786+0000","last_clean":"2023-10-18T16:07:41.799786+0000","last_became_active":"2023-10-18T16:07:41.438408+0000","last_became_peered":"2023-10-18T16:07:41.438408+0000","last_unstale":"2023-10-18T16:07:41.799786+0000","last_undegraded":"2023-10-18T16:07:41.799786+0000","last_fullsized":"2023-10-18T16:07:41.799786+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.806161+0000","last_change":"2023-10-18T16:07:41.800718+0000","last_active":"2023-10-18T16:07:41.806161+0000","last_peered":"2023-10-18T16:07:41.806161+0000","last_clean":"2023-10-18T16:07:41.806161+0000","last_became_active":"2023-10-18T16:07:41.800400+0000","last_became_peered":"2023-10-18T16:07:41.800400+0000","last_unstale":"2023-10-18T16:07:41.806161+0000","last_undegraded":"2023-10-18T16:07:41.806161+0000","last_fullsized":"2023-10-18T16:07:41.806161+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.446623+0000","last_change":"2023-10-18T16:07:41.438151+0000","last_active":"2023-10-18T16:07:41.446623+0000","last_peered":"2023-10-18T16:07:41.446623+0000","last_clean":"2023-10-18T16:07:41.446623+0000","last_became_active":"2023-10-18T16:07:41.437504+0000","last_became_peered":"2023-10-18T16:07:41.437504+0000","last_unstale":"2023-10-18T16:07:41.446623+0000","last_undegraded":"2023-10-18T16:07:41.446623+0000","last_fullsized":"2023-10-18T16:07:41.446623+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":8,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.802411+0000","last_change":"2023-10-18T16:07:41.802411+0000","last_active":"2023-10-18T16:07:41.802411+0000","last_peered":"2023-10-18T16:07:41.802411+0000","last_clean":"2023-10-18T16:07:41.802411+0000","last_became_active":"2023-10-18T16:07:41.802124+0000","last_became_peered":"2023-10-18T16:07:41.802124+0000","last_unstale":"2023-10-18T16:07:41.802411+0000","last_undegraded":"2023-10-18T16:07:41.802411+0000","last_fullsized":"2023-10-18T16:07:41.802411+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.797104+0000","last_change":"2023-10-18T16:07:41.439071+0000","last_active":"2023-10-18T16:07:41.797104+0000","last_peered":"2023-10-18T16:07:41.797104+0000","last_clean":"2023-10-18T16:07:41.797104+0000","last_became_active":"2023-10-18T16:07:41.438144+0000","last_became_peered":"2023-10-18T16:07:41.438144+0000","last_unstale":"2023-10-18T16:07:41.797104+0000","last_undegraded":"2023-10-18T16:07:41.797104+0000","last_fullsized":"2023-10-18T16:07:41.797104+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":8,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.798403+0000","last_change":"2023-10-18T16:07:41.798403+0000","last_active":"2023-10-18T16:07:41.798403+0000","last_peered":"2023-10-18T16:07:41.798403+0000","last_clean":"2023-10-18T16:07:41.798403+0000","last_became_active":"2023-10-18T16:07:41.798149+0000","last_became_peered":"2023-10-18T16:07:41.798149+0000","last_unstale":"2023-10-18T16:07:41.798403+0000","last_undegraded":"2023-10-18T16:07:41.798403+0000","last_fullsized":"2023-10-18T16:07:41.798403+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.438801+0000","last_change":"2023-10-18T16:07:41.432914+0000","last_active":"2023-10-18T16:07:41.438801+0000","last_peered":"2023-10-18T16:07:41.438801+0000","last_clean":"2023-10-18T16:07:41.438801+0000","last_became_active":"2023-10-18T16:07:41.432601+0000","last_became_peered":"2023-10-18T16:07:41.432601+0000","last_unstale":"2023-10-18T16:07:41.438801+0000","last_undegraded":"2023-10-18T16:07:41.438801+0000","last_fullsized":"2023-10-18T16:07:41.438801+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.806049+0000","last_change":"2023-10-18T16:07:41.801443+0000","last_active":"2023-10-18T16:07:41.806049+0000","last_peered":"2023-10-18T16:07:41.806049+0000","last_clean":"2023-10-18T16:07:41.806049+0000","last_became_active":"2023-10-18T16:07:41.801271+0000","last_became_peered":"2023-10-18T16:07:41.801271+0000","last_unstale":"2023-10-18T16:07:41.806049+0000","last_undegraded":"2023-10-18T16:07:41.806049+0000","last_fullsized":"2023-10-18T16:07:41.806049+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.436412+0000","last_change":"2023-10-18T16:07:41.434525+0000","last_active":"2023-10-18T16:07:41.436412+0000","last_peered":"2023-10-18T16:07:41.436412+0000","last_clean":"2023-10-18T16:07:41.436412+0000","last_became_active":"2023-10-18T16:07:41.434220+0000","last_became_peered":"2023-10-18T16:07:41.434220+0000","last_unstale":"2023-10-18T16:07:41.436412+0000","last_undegraded":"2023-10-18T16:07:41.436412+0000","last_fullsized":"2023-10-18T16:07:41.436412+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.808917+0000","last_change":"2023-10-18T16:07:41.807740+0000","last_active":"2023-10-18T16:07:41.808917+0000","last_peered":"2023-10-18T16:07:41.808917+0000","last_clean":"2023-10-18T16:07:41.808917+0000","last_became_active":"2023-10-18T16:07:41.807473+0000","last_became_peered":"2023-10-18T16:07:41.807473+0000","last_unstale":"2023-10-18T16:07:41.808917+0000","last_undegraded":"2023-10-18T16:07:41.808917+0000","last_fullsized":"2023-10-18T16:07:41.808917+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.433447+0000","last_change":"2023-10-18T16:07:41.427326+0000","last_active":"2023-10-18T16:07:41.433447+0000","last_peered":"2023-10-18T16:07:41.433447+0000","last_clean":"2023-10-18T16:07:41.433447+0000","last_became_active":"2023-10-18T16:07:41.427083+0000","last_became_peered":"2023-10-18T16:07:41.427083+0000","last_unstale":"2023-10-18T16:07:41.433447+0000","last_undegraded":"2023-10-18T16:07:41.433447+0000","last_fullsized":"2023-10-18T16:07:41.433447+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":8,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.804453+0000","last_change":"2023-10-18T16:07:41.804453+0000","last_active":"2023-10-18T16:07:41.804453+0000","last_peered":"2023-10-18T16:07:41.804453+0000","last_clean":"2023-10-18T16:07:41.804453+0000","last_became_active":"2023-10-18T16:07:41.802577+0000","last_became_peered":"2023-10-18T16:07:41.802577+0000","last_unstale":"2023-10-18T16:07:41.804453+0000","last_undegraded":"2023-10-18T16:07:41.804453+0000","last_fullsized":"2023-10-18T16:07:41.804453+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.798474+0000","last_change":"2023-10-18T16:07:41.436757+0000","last_active":"2023-10-18T16:07:41.798474+0000","last_peered":"2023-10-18T16:07:41.798474+0000","last_clean":"2023-10-18T16:07:41.798474+0000","last_became_active":"2023-10-18T16:07:41.436423+0000","last_became_peered":"2023-10-18T16:07:41.436423+0000","last_unstale":"2023-10-18T16:07:41.798474+0000","last_undegraded":"2023-10-18T16:07:41.798474+0000","last_fullsized":"2023-10-18T16:07:41.798474+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574851,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297320,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074520,"statfs":{"total":96636764160,"available":96332308480,"internally_reserved":0,"allocated":229376,"data_stored":85341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53700000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52400000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68899999999999995}]}]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297308,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074532,"statfs":{"total":96636764160,"available":96332320768,"internally_reserved":0,"allocated":217088,"data_stored":78637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66300000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65000000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51300000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574851,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297308,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074532,"statfs":{"total":96636764160,"available":96332320768,"internally_reserved":0,"allocated":217088,"data_stored":78637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.318}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.373}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574851,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297308,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074532,"statfs":{"total":96636764160,"available":96332320768,"internally_reserved":0,"allocated":217088,"data_stored":78637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28899999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65900000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56799999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60899999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574851,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297320,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074520,"statfs":{"total":96636764160,"available":96332308480,"internally_reserved":0,"allocated":229376,"data_stored":85341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34200000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36799999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47999999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66300000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63700000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574852,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297308,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074532,"statfs":{"total":96636764160,"available":96332320768,"internally_reserved":0,"allocated":217088,"data_stored":78637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34599999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35899999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58599999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68999999999999995}]}]},{"osd":4,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297320,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074520,"statfs":{"total":96636764160,"available":96332308480,"internally_reserved":0,"allocated":229376,"data_stored":85341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58699999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69099999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42799999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38500000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]}]},{"osd":5,"up_from":13,"seq":55834574851,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297308,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074532,"statfs":{"total":96636764160,"available":96332320768,"internally_reserved":0,"allocated":217088,"data_stored":78637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70299999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68999999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53300000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-18T16:07:47.760 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-18T16:07:47.760 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-10-18T16:07:47.760 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-10-18T16:07:47.761 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-10-18T16:07:47.761 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-18T16:07:47.906 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.902+0000 7f346e927700 1 Processor -- start 2023-10-18T16:07:47.906 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.902+0000 7f346e927700 1 -- start start 2023-10-18T16:07:47.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.903+0000 7f346e927700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f346814d750 0x7f3468150350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:47.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.903+0000 7f346e927700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3468066e40 0x7f3468068dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:47.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.903+0000 7f346e927700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3468068940 con 0x7f346814db30 2023-10-18T16:07:47.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.903+0000 7f346e927700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3468067260 con 0x7f3468066e40 2023-10-18T16:07:47.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.903+0000 7f346e927700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3468067460 con 0x7f346814d750 2023-10-18T16:07:47.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.903+0000 7f346cec4700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3468066e40 0x7f3468068dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:47.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.903+0000 7f346cec4700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3468066e40 0x7f3468068dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59440/0 (socket says 172.21.15.88:59440) 2023-10-18T16:07:47.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.903+0000 7f346cec4700 1 -- 172.21.15.88:0/1626225665 learned_addr learned my addr 172.21.15.88:0/1626225665 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:47.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.903+0000 7f34677fe700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f346814db30 0x7f3468063760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:52452/0 (socket says 172.21.15.88:52452) 2023-10-18T16:07:47.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.903+0000 7f3467fff700 1 --2- 172.21.15.88:0/1626225665 >> v2:172.21.15.151:3300/0 conn(0x7f346814d750 0x7f3468150350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:47.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.904+0000 7f346cec4700 1 -- 172.21.15.88:0/1626225665 >> v2:172.21.15.151:3300/0 conn(0x7f346814d750 msgr2=0x7f3468150350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:47.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.904+0000 7f346cec4700 1 --2- 172.21.15.88:0/1626225665 >> v2:172.21.15.151:3300/0 conn(0x7f346814d750 0x7f3468150350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:47.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.904+0000 7f346cec4700 1 -- 172.21.15.88:0/1626225665 >> v1:172.21.15.88:6789/0 conn(0x7f346814db30 legacy=0x7f3468063760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:47.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.904+0000 7f346cec4700 1 -- 172.21.15.88:0/1626225665 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3468067740 con 0x7f3468066e40 2023-10-18T16:07:47.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.904+0000 7f3466ffd700 1 -- 172.21.15.88:0/1626225665 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 829933530 0 0) 0x7f3468068940 con 0x7f346814db30 2023-10-18T16:07:47.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.904+0000 7f346cec4700 1 --2- 172.21.15.88:0/1626225665 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3468066e40 0x7f3468068dd0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f3468065e00 tx=0x7f3460009840).ready entity=mon.0 client_cookie=492c973f784a8883 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:47.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.904+0000 7f3466ffd700 1 -- 172.21.15.88:0/1626225665 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f346000e070 con 0x7f3468066e40 2023-10-18T16:07:47.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.904+0000 7f3466ffd700 1 -- 172.21.15.88:0/1626225665 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f346000ba50 con 0x7f3468066e40 2023-10-18T16:07:47.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.904+0000 7f3466ffd700 1 -- 172.21.15.88:0/1626225665 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f346000bdd0 con 0x7f3468066e40 2023-10-18T16:07:47.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.904+0000 7f346e927700 1 -- 172.21.15.88:0/1626225665 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3468066e40 msgr2=0x7f3468068dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:47.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.904+0000 7f346e927700 1 --2- 172.21.15.88:0/1626225665 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3468066e40 0x7f3468068dd0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f3468065e00 tx=0x7f3460009840).stop 2023-10-18T16:07:47.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.904+0000 7f346e927700 1 -- 172.21.15.88:0/1626225665 shutdown_connections 2023-10-18T16:07:47.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.904+0000 7f346e927700 1 --2- 172.21.15.88:0/1626225665 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3468066e40 0x7f3468068dd0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:47.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.904+0000 7f346e927700 1 --2- 172.21.15.88:0/1626225665 >> v2:172.21.15.151:3300/0 conn(0x7f346814d750 0x7f3468150350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:47.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.904+0000 7f346e927700 1 -- 172.21.15.88:0/1626225665 >> 172.21.15.88:0/1626225665 conn(0x7f34680c48f0 msgr2=0x7f34680c4d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:47.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.904+0000 7f346e927700 1 -- 172.21.15.88:0/1626225665 shutdown_connections 2023-10-18T16:07:47.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.904+0000 7f346e927700 1 -- 172.21.15.88:0/1626225665 wait complete. 2023-10-18T16:07:47.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.905+0000 7f346e927700 1 Processor -- start 2023-10-18T16:07:47.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.906+0000 7f346e927700 1 -- start start 2023-10-18T16:07:47.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.906+0000 7f346e927700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3468066e40 0x7f34680bb080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:47.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.906+0000 7f346e927700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f346814d750 0x7f346814fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:47.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.906+0000 7f346e927700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f346814db30 0x7f3468151d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:47.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.906+0000 7f346e927700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34680babf0 con 0x7f346814db30 2023-10-18T16:07:47.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.906+0000 7f346e927700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34681642e0 con 0x7f346814d750 2023-10-18T16:07:47.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.906+0000 7f346e927700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34681645c0 con 0x7f3468066e40 2023-10-18T16:07:47.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.906+0000 7f346cec4700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f346814db30 0x7f3468151d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:47.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.906+0000 7f346cec4700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f346814db30 0x7f3468151d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59452/0 (socket says 172.21.15.88:59452) 2023-10-18T16:07:47.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.906+0000 7f346cec4700 1 -- 172.21.15.88:0/2164886753 learned_addr learned my addr 172.21.15.88:0/2164886753 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:47.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.907+0000 7f34677fe700 1 --2- 172.21.15.88:0/2164886753 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f346814d750 0x7f346814fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:47.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.907+0000 7f3467fff700 1 --2- 172.21.15.88:0/2164886753 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3468066e40 0x7f34680bb080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:47.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.907+0000 7f346cec4700 1 -- 172.21.15.88:0/2164886753 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3468066e40 msgr2=0x7f34680bb080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:47.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.907+0000 7f346cec4700 1 --2- 172.21.15.88:0/2164886753 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3468066e40 0x7f34680bb080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:47.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.907+0000 7f346cec4700 1 -- 172.21.15.88:0/2164886753 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f346814d750 msgr2=0x7f346814fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:47.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.907+0000 7f346cec4700 1 --2- 172.21.15.88:0/2164886753 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f346814d750 0x7f346814fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:47.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.907+0000 7f346cec4700 1 -- 172.21.15.88:0/2164886753 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3460009ae0 con 0x7f346814db30 2023-10-18T16:07:47.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.907+0000 7f3467fff700 1 --2- 172.21.15.88:0/2164886753 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3468066e40 0x7f34680bb080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:07:47.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.907+0000 7f346cec4700 1 --2- 172.21.15.88:0/2164886753 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f346814db30 0x7f3468151d30 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f3460009880 tx=0x7f3468065180).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:47.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.907+0000 7f344ffff700 1 -- 172.21.15.88:0/2164886753 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f346000b6c0 con 0x7f346814db30 2023-10-18T16:07:47.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.907+0000 7f344ffff700 1 -- 172.21.15.88:0/2164886753 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34600092e0 con 0x7f346814db30 2023-10-18T16:07:47.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.907+0000 7f346e927700 1 -- 172.21.15.88:0/2164886753 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34681648a0 con 0x7f346814db30 2023-10-18T16:07:47.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.907+0000 7f344ffff700 1 -- 172.21.15.88:0/2164886753 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f346001d620 con 0x7f346814db30 2023-10-18T16:07:47.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.907+0000 7f346e927700 1 -- 172.21.15.88:0/2164886753 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34680ba860 con 0x7f346814db30 2023-10-18T16:07:47.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.908+0000 7f346e927700 1 -- 172.21.15.88:0/2164886753 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34680bad60 con 0x7f346814db30 2023-10-18T16:07:47.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.908+0000 7f344ffff700 1 -- 172.21.15.88:0/2164886753 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f346001d9d0 con 0x7f346814db30 2023-10-18T16:07:47.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.908+0000 7f344ffff700 1 --2- 172.21.15.88:0/2164886753 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f345405f150 0x7f3454061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:47.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.909+0000 7f344ffff700 1 -- 172.21.15.88:0/2164886753 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6121+0+0 (secure 0 0 0) 0x7f346007bde0 con 0x7f346814db30 2023-10-18T16:07:47.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.909+0000 7f3467fff700 1 --2- 172.21.15.88:0/2164886753 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f345405f150 0x7f3454061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:47.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.909+0000 7f3467fff700 1 --2- 172.21.15.88:0/2164886753 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f345405f150 0x7f3454061610 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f345800bb60 tx=0x7f3458008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:47.923 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:47.913+0000 7f344ffff700 1 -- 172.21.15.88:0/2164886753 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f346004d0b0 con 0x7f346814db30 2023-10-18T16:07:48.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.120+0000 7f346e927700 1 -- 172.21.15.88:0/2164886753 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f3468067a20 con 0x7f346814db30 2023-10-18T16:07:48.126 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.123+0000 7f344ffff700 1 -- 172.21.15.88:0/2164886753 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v4) v1 ==== 74+0+148955 (secure 0 0 0) 0x7f34600506d0 con 0x7f346814db30 2023-10-18T16:07:48.128 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:07:48.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.125+0000 7f346e927700 1 -- 172.21.15.88:0/2164886753 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f345405f150 msgr2=0x7f3454061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:48.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.125+0000 7f346e927700 1 --2- 172.21.15.88:0/2164886753 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f345405f150 0x7f3454061610 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f345800bb60 tx=0x7f3458008040).stop 2023-10-18T16:07:48.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.125+0000 7f346e927700 1 -- 172.21.15.88:0/2164886753 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f346814db30 msgr2=0x7f3468151d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:48.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.125+0000 7f346e927700 1 --2- 172.21.15.88:0/2164886753 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f346814db30 0x7f3468151d30 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f3460009880 tx=0x7f3468065180).stop 2023-10-18T16:07:48.131 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.125+0000 7f346e927700 1 -- 172.21.15.88:0/2164886753 shutdown_connections 2023-10-18T16:07:48.131 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.125+0000 7f346e927700 1 --2- 172.21.15.88:0/2164886753 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f345405f150 0x7f3454061610 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:48.131 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.125+0000 7f346e927700 1 --2- 172.21.15.88:0/2164886753 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f346814db30 0x7f3468151d30 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:48.131 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.125+0000 7f346e927700 1 --2- 172.21.15.88:0/2164886753 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3468066e40 0x7f34680bb080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:48.131 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.125+0000 7f346e927700 1 --2- 172.21.15.88:0/2164886753 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f346814d750 0x7f346814fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:48.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.125+0000 7f346e927700 1 -- 172.21.15.88:0/2164886753 >> 172.21.15.88:0/2164886753 conn(0x7f34680c48f0 msgr2=0x7f34680c39a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:48.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.125+0000 7f346e927700 1 -- 172.21.15.88:0/2164886753 shutdown_connections 2023-10-18T16:07:48.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.125+0000 7f346e927700 1 -- 172.21.15.88:0/2164886753 wait complete. 2023-10-18T16:07:48.141 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":4,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6832","nonce":90916},{"type":"v1","addr":"172.21.15.151:6833","nonce":90916}]},"active_addr":"172.21.15.151:6833/90916","active_change":"2023-10-18T16:07:28.447838+0000","active_mgr_features":4540138314316775423,"available":true,"standbys":[{"gid":4107,"name":"y","mgr_features":4540138314316775423,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.151:0","nonce":4043993804}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.151:0","nonce":1121528681}]}]} 2023-10-18T16:07:48.144 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-10-18T16:07:48.144 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-10-18T16:07:48.145 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-18T16:07:48.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.253+0000 7fe49b59e700 1 Processor -- start 2023-10-18T16:07:48.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.253+0000 7fe49b59e700 1 -- start start 2023-10-18T16:07:48.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.254+0000 7fe49b59e700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe49c064360 0x7fe49c1538a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:48.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.254+0000 7fe49b59e700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe49c151840 con 0x7fe49c14d7c0 2023-10-18T16:07:48.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.254+0000 7fe49b59e700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe49c0c42d0 con 0x7fe49c14dba0 2023-10-18T16:07:48.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.254+0000 7fe49b59e700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe49c064830 con 0x7fe49c064360 2023-10-18T16:07:48.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.254+0000 7fe49a59c700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fe49c14d7c0 0x7fe49c0a8de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:52458/0 (socket says 172.21.15.88:52458) 2023-10-18T16:07:48.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.254+0000 7fe49a59c700 1 -- 172.21.15.88:0/2426150175 learned_addr learned my addr 172.21.15.88:0/2426150175 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:07:48.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.254+0000 7fe491d9b700 1 --2- 172.21.15.88:0/2426150175 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe49c064360 0x7fe49c1538a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:48.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.254+0000 7fe499d9b700 1 -- 172.21.15.88:0/2426150175 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3223283497 0 0) 0x7fe49c151840 con 0x7fe49c14d7c0 2023-10-18T16:07:48.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.254+0000 7fe499d9b700 1 -- 172.21.15.88:0/2426150175 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4880033f0 con 0x7fe49c14d7c0 2023-10-18T16:07:48.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.254+0000 7fe499d9b700 1 -- 172.21.15.88:0/2426150175 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1652519049 0 0) 0x7fe49c0c42d0 con 0x7fe49c14dba0 2023-10-18T16:07:48.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.254+0000 7fe499d9b700 1 -- 172.21.15.88:0/2426150175 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe49c151840 con 0x7fe49c14dba0 2023-10-18T16:07:48.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.254+0000 7fe499d9b700 1 -- 172.21.15.88:0/2426150175 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3111323606 0 0) 0x7fe4880033f0 con 0x7fe49c14d7c0 2023-10-18T16:07:48.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.254+0000 7fe491d9b700 1 -- 172.21.15.88:0/2426150175 >> v1:172.21.15.151:6789/0 conn(0x7fe49c14dba0 legacy=0x7fe49c151cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:48.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.254+0000 7fe491d9b700 1 -- 172.21.15.88:0/2426150175 >> v1:172.21.15.88:6789/0 conn(0x7fe49c14d7c0 legacy=0x7fe49c0a8de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:48.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.254+0000 7fe491d9b700 1 -- 172.21.15.88:0/2426150175 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe49c064b10 con 0x7fe49c064360 2023-10-18T16:07:48.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.255+0000 7fe491d9b700 1 --2- 172.21.15.88:0/2426150175 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe49c064360 0x7fe49c1538a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fe484009790 tx=0x7fe484009060).ready entity=mon.2 client_cookie=8513e1faa464d5f5 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:48.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.255+0000 7fe499d9b700 1 -- 172.21.15.88:0/2426150175 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe48c001470 con 0x7fe49c064360 2023-10-18T16:07:48.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.255+0000 7fe499d9b700 1 -- 172.21.15.88:0/2426150175 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe484011400 con 0x7fe49c064360 2023-10-18T16:07:48.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.255+0000 7fe499d9b700 1 -- 172.21.15.88:0/2426150175 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe484011780 con 0x7fe49c064360 2023-10-18T16:07:48.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.255+0000 7fe49b59e700 1 -- 172.21.15.88:0/2426150175 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe49c064360 msgr2=0x7fe49c1538a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:48.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.255+0000 7fe49b59e700 1 --2- 172.21.15.88:0/2426150175 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe49c064360 0x7fe49c1538a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fe484009790 tx=0x7fe484009060).stop 2023-10-18T16:07:48.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.255+0000 7fe49b59e700 1 -- 172.21.15.88:0/2426150175 shutdown_connections 2023-10-18T16:07:48.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.255+0000 7fe49b59e700 1 --2- 172.21.15.88:0/2426150175 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe49c064360 0x7fe49c1538a0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:48.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.255+0000 7fe49b59e700 1 -- 172.21.15.88:0/2426150175 >> 172.21.15.88:0/2426150175 conn(0x7fe49c069d80 msgr2=0x7fe49c06a190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:48.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.255+0000 7fe49b59e700 1 -- 172.21.15.88:0/2426150175 shutdown_connections 2023-10-18T16:07:48.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.255+0000 7fe49b59e700 1 -- 172.21.15.88:0/2426150175 wait complete. 2023-10-18T16:07:48.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.256+0000 7fe49b59e700 1 Processor -- start 2023-10-18T16:07:48.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.256+0000 7fe49b59e700 1 -- start start 2023-10-18T16:07:48.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.257+0000 7fe49b59e700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe49c064360 0x7fe49c0aa300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:48.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.257+0000 7fe49b59e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe49c14d7c0 0x7fe49c163c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:48.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.257+0000 7fe49b59e700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe49c14dba0 0x7fe49c164130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:48.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.257+0000 7fe49b59e700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe49c0bc130 con 0x7fe49c14dba0 2023-10-18T16:07:48.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.257+0000 7fe49b59e700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe49c0aa7f0 con 0x7fe49c14d7c0 2023-10-18T16:07:48.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.257+0000 7fe49b59e700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe49c1647a0 con 0x7fe49c064360 2023-10-18T16:07:48.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.257+0000 7fe49ad9d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe49c14dba0 0x7fe49c164130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:48.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.257+0000 7fe49ad9d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe49c14dba0 0x7fe49c164130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59462/0 (socket says 172.21.15.88:59462) 2023-10-18T16:07:48.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.257+0000 7fe49ad9d700 1 -- 172.21.15.88:0/730856152 learned_addr learned my addr 172.21.15.88:0/730856152 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:48.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.257+0000 7fe49a59c700 1 --2- 172.21.15.88:0/730856152 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe49c064360 0x7fe49c0aa300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:48.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.257+0000 7fe491d9b700 1 --2- 172.21.15.88:0/730856152 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe49c14d7c0 0x7fe49c163c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:48.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.257+0000 7fe49ad9d700 1 -- 172.21.15.88:0/730856152 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe49c064360 msgr2=0x7fe49c0aa300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:48.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.258+0000 7fe49ad9d700 1 --2- 172.21.15.88:0/730856152 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe49c064360 0x7fe49c0aa300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:48.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.258+0000 7fe49ad9d700 1 -- 172.21.15.88:0/730856152 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe49c14d7c0 msgr2=0x7fe49c163c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:48.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.258+0000 7fe49ad9d700 1 --2- 172.21.15.88:0/730856152 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe49c14d7c0 0x7fe49c163c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:48.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.258+0000 7fe49ad9d700 1 -- 172.21.15.88:0/730856152 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4840092a0 con 0x7fe49c14dba0 2023-10-18T16:07:48.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.258+0000 7fe49a59c700 1 --2- 172.21.15.88:0/730856152 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe49c064360 0x7fe49c0aa300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:07:48.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.258+0000 7fe491d9b700 1 --2- 172.21.15.88:0/730856152 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe49c14d7c0 0x7fe49c163c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:07:48.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.258+0000 7fe49ad9d700 1 --2- 172.21.15.88:0/730856152 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe49c14dba0 0x7fe49c164130 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fe49400c790 tx=0x7fe494007f20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:48.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.258+0000 7fe492ffd700 1 -- 172.21.15.88:0/730856152 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe494010510 con 0x7fe49c14dba0 2023-10-18T16:07:48.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.258+0000 7fe492ffd700 1 -- 172.21.15.88:0/730856152 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe494010670 con 0x7fe49c14dba0 2023-10-18T16:07:48.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.258+0000 7fe49b59e700 1 -- 172.21.15.88:0/730856152 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe49c0bc3c0 con 0x7fe49c14dba0 2023-10-18T16:07:48.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.258+0000 7fe492ffd700 1 -- 172.21.15.88:0/730856152 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe494010a30 con 0x7fe49c14dba0 2023-10-18T16:07:48.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.258+0000 7fe49b59e700 1 -- 172.21.15.88:0/730856152 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe49c0bc980 con 0x7fe49c14dba0 2023-10-18T16:07:48.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.259+0000 7fe49b59e700 1 -- 172.21.15.88:0/730856152 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe49c0bcb70 con 0x7fe49c14dba0 2023-10-18T16:07:48.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.259+0000 7fe492ffd700 1 -- 172.21.15.88:0/730856152 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7fe49401b400 con 0x7fe49c14dba0 2023-10-18T16:07:48.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.259+0000 7fe492ffd700 1 --2- 172.21.15.88:0/730856152 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fe47c05f150 0x7fe47c061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:48.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.260+0000 7fe492ffd700 1 -- 172.21.15.88:0/730856152 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6121+0+0 (secure 0 0 0) 0x7fe49407a1b0 con 0x7fe49c14dba0 2023-10-18T16:07:48.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.260+0000 7fe49a59c700 1 --2- 172.21.15.88:0/730856152 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fe47c05f150 0x7fe47c061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:48.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.260+0000 7fe49a59c700 1 --2- 172.21.15.88:0/730856152 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fe47c05f150 0x7fe47c061610 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fe48c0011e0 tx=0x7fe48c000bc0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:48.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.264+0000 7fe492ffd700 1 -- 172.21.15.88:0/730856152 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7fe49404b360 con 0x7fe49c14dba0 2023-10-18T16:07:48.430 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.427+0000 7fe49b59e700 1 -- 172.21.15.88:0/730856152 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe49c083b10 con 0x7fe49c14dba0 2023-10-18T16:07:48.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.428+0000 7fe492ffd700 1 -- 172.21.15.88:0/730856152 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16928 (secure 0 0 0) 0x7fe49404e980 con 0x7fe49c14dba0 2023-10-18T16:07:48.432 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:07:48.432 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":22,"fsid":"fb1f77e8-ead0-4f90-8128-fdf5137eb16f","created":"2023-10-18T16:07:23.662468+0000","modified":"2023-10-18T16:07:42.784120+0000","last_up_change":"2023-10-18T16:07:32.498203+0000","last_in_change":"2023-10-18T16:07:26.064784+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-10-18T16:07:32.536320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-18T16:07:34.769989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-10-18T16:07:39.439702+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-10-18T16:07:40.753774+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e421c20c-f7bc-4574-97a2-201bb4a16b4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":91007},{"type":"v1","addr":"172.21.15.88:6809","nonce":91007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":91007},{"type":"v1","addr":"172.21.15.88:6811","nonce":91007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":91007},{"type":"v1","addr":"172.21.15.88:6815","nonce":91007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":91007},{"type":"v1","addr":"172.21.15.88:6813","nonce":91007}]},"public_addr":"172.21.15.88:6809/91007","cluster_addr":"172.21.15.88:6811/91007","heartbeat_back_addr":"172.21.15.88:6815/91007","heartbeat_front_addr":"172.21.15.88:6813/91007","state":["exists","up"]},{"osd":1,"uuid":"9485c5ec-379a-4038-a2d8-1f1081fe2eea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":91009},{"type":"v1","addr":"172.21.15.88:6825","nonce":91009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":91009},{"type":"v1","addr":"172.21.15.88:6827","nonce":91009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":91009},{"type":"v1","addr":"172.21.15.88:6831","nonce":91009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":91009},{"type":"v1","addr":"172.21.15.88:6829","nonce":91009}]},"public_addr":"172.21.15.88:6825/91009","cluster_addr":"172.21.15.88:6827/91009","heartbeat_back_addr":"172.21.15.88:6831/91009","heartbeat_front_addr":"172.21.15.88:6829/91009","state":["exists","up"]},{"osd":2,"uuid":"671895e2-57f3-4f1a-ba4a-a580af1f3ec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":91005},{"type":"v1","addr":"172.21.15.88:6801","nonce":91005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":91005},{"type":"v1","addr":"172.21.15.88:6803","nonce":91005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":91005},{"type":"v1","addr":"172.21.15.88:6807","nonce":91005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":91005},{"type":"v1","addr":"172.21.15.88:6805","nonce":91005}]},"public_addr":"172.21.15.88:6801/91005","cluster_addr":"172.21.15.88:6803/91005","heartbeat_back_addr":"172.21.15.88:6807/91005","heartbeat_front_addr":"172.21.15.88:6805/91005","state":["exists","up"]},{"osd":3,"uuid":"0c4e6e7f-7d9c-4e65-beec-fd0ca384f320","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":91008},{"type":"v1","addr":"172.21.15.88:6817","nonce":91008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":91008},{"type":"v1","addr":"172.21.15.88:6819","nonce":91008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":91008},{"type":"v1","addr":"172.21.15.88:6823","nonce":91008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":91008},{"type":"v1","addr":"172.21.15.88:6821","nonce":91008}]},"public_addr":"172.21.15.88:6817/91008","cluster_addr":"172.21.15.88:6819/91008","heartbeat_back_addr":"172.21.15.88:6823/91008","heartbeat_front_addr":"172.21.15.88:6821/91008","state":["exists","up"]},{"osd":4,"uuid":"29570691-ef35-4b80-9247-aa3703877af4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":91514},{"type":"v1","addr":"172.21.15.151:6801","nonce":91514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":91514},{"type":"v1","addr":"172.21.15.151:6803","nonce":91514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":91514},{"type":"v1","addr":"172.21.15.151:6807","nonce":91514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":91514},{"type":"v1","addr":"172.21.15.151:6805","nonce":91514}]},"public_addr":"172.21.15.151:6801/91514","cluster_addr":"172.21.15.151:6803/91514","heartbeat_back_addr":"172.21.15.151:6807/91514","heartbeat_front_addr":"172.21.15.151:6805/91514","state":["exists","up"]},{"osd":5,"uuid":"26540b7b-032a-4b3b-bf6c-b131a449fa99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":91513},{"type":"v1","addr":"172.21.15.151:6809","nonce":91513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":91513},{"type":"v1","addr":"172.21.15.151:6811","nonce":91513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":91513},{"type":"v1","addr":"172.21.15.151:6815","nonce":91513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":91513},{"type":"v1","addr":"172.21.15.151:6813","nonce":91513}]},"public_addr":"172.21.15.151:6809/91513","cluster_addr":"172.21.15.151:6811/91513","heartbeat_back_addr":"172.21.15.151:6815/91513","heartbeat_front_addr":"172.21.15.151:6813/91513","state":["exists","up"]},{"osd":6,"uuid":"7af01f9c-90db-45fd-93c8-09517c9b0881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":91544},{"type":"v1","addr":"172.21.15.151:6817","nonce":91544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":91544},{"type":"v1","addr":"172.21.15.151:6819","nonce":91544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":91544},{"type":"v1","addr":"172.21.15.151:6823","nonce":91544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":91544},{"type":"v1","addr":"172.21.15.151:6821","nonce":91544}]},"public_addr":"172.21.15.151:6817/91544","cluster_addr":"172.21.15.151:6819/91544","heartbeat_back_addr":"172.21.15.151:6823/91544","heartbeat_front_addr":"172.21.15.151:6821/91544","state":["exists","up"]},{"osd":7,"uuid":"b527b0a5-4249-4854-a30f-bff897757239","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":91545},{"type":"v1","addr":"172.21.15.151:6825","nonce":91545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":91545},{"type":"v1","addr":"172.21.15.151:6827","nonce":91545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":91545},{"type":"v1","addr":"172.21.15.151:6831","nonce":91545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":91545},{"type":"v1","addr":"172.21.15.151:6829","nonce":91545}]},"public_addr":"172.21.15.151:6825/91545","cluster_addr":"172.21.15.151:6827/91545","heartbeat_back_addr":"172.21.15.151:6831/91545","heartbeat_front_addr":"172.21.15.151:6829/91545","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T16:07:31.216742+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T16:07:31.467229+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T16:07:31.220809+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T16:07:31.414999+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T16:07:31.188976+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T16:07:31.252153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T16:07:31.254324+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T16:07:31.203434+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-18T16:07:48.434 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.430+0000 7fe49b59e700 1 -- 172.21.15.88:0/730856152 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fe47c05f150 msgr2=0x7fe47c061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:48.434 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.430+0000 7fe49b59e700 1 --2- 172.21.15.88:0/730856152 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fe47c05f150 0x7fe47c061610 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fe48c0011e0 tx=0x7fe48c000bc0).stop 2023-10-18T16:07:48.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.430+0000 7fe49b59e700 1 -- 172.21.15.88:0/730856152 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe49c14dba0 msgr2=0x7fe49c164130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:48.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.430+0000 7fe49b59e700 1 --2- 172.21.15.88:0/730856152 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe49c14dba0 0x7fe49c164130 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fe49400c790 tx=0x7fe494007f20).stop 2023-10-18T16:07:48.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.430+0000 7fe49b59e700 1 -- 172.21.15.88:0/730856152 shutdown_connections 2023-10-18T16:07:48.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.430+0000 7fe49b59e700 1 --2- 172.21.15.88:0/730856152 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fe47c05f150 0x7fe47c061610 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:48.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.430+0000 7fe49b59e700 1 --2- 172.21.15.88:0/730856152 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe49c14dba0 0x7fe49c164130 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:48.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.430+0000 7fe49b59e700 1 --2- 172.21.15.88:0/730856152 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe49c064360 0x7fe49c0aa300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:48.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.430+0000 7fe49b59e700 1 --2- 172.21.15.88:0/730856152 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe49c14d7c0 0x7fe49c163c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:48.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.430+0000 7fe49b59e700 1 -- 172.21.15.88:0/730856152 >> 172.21.15.88:0/730856152 conn(0x7fe49c069d80 msgr2=0x7fe49c06a160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:48.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.430+0000 7fe49b59e700 1 -- 172.21.15.88:0/730856152 shutdown_connections 2023-10-18T16:07:48.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.430+0000 7fe49b59e700 1 -- 172.21.15.88:0/730856152 wait complete. 2023-10-18T16:07:48.446 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-10-18T16:07:48.446 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-18T16:07:48.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.556+0000 7f9bf4720700 1 Processor -- start 2023-10-18T16:07:48.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.556+0000 7f9bf4720700 1 -- start start 2023-10-18T16:07:48.562 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.557+0000 7f9bf4720700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9bec14b480 0x7f9bec0636a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:48.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.557+0000 7f9bf4720700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f9bec064f80 0x7f9bec063b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:48.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.557+0000 7f9bf4720700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bec0becd0 con 0x7f9bec14e100 2023-10-18T16:07:48.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.557+0000 7f9bf4720700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bec14e590 con 0x7f9bec14b480 2023-10-18T16:07:48.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.557+0000 7f9bf4720700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bec14e870 con 0x7f9bec064f80 2023-10-18T16:07:48.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.557+0000 7f9bf24bc700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9bec14b480 0x7f9bec0636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:48.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.557+0000 7f9bf24bc700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9bec14b480 0x7f9bec0636a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59470/0 (socket says 172.21.15.88:59470) 2023-10-18T16:07:48.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.557+0000 7f9bf24bc700 1 -- 172.21.15.88:0/1640086654 learned_addr learned my addr 172.21.15.88:0/1640086654 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:48.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.557+0000 7f9bf2cbd700 1 --2- 172.21.15.88:0/1640086654 >> v2:172.21.15.151:3300/0 conn(0x7f9bec064f80 0x7f9bec063b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:48.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.557+0000 7f9bf24bc700 1 -- 172.21.15.88:0/1640086654 >> v2:172.21.15.151:3300/0 conn(0x7f9bec064f80 msgr2=0x7f9bec063b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:48.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.557+0000 7f9bf24bc700 1 --2- 172.21.15.88:0/1640086654 >> v2:172.21.15.151:3300/0 conn(0x7f9bec064f80 0x7f9bec063b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:48.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.557+0000 7f9bf24bc700 1 -- 172.21.15.88:0/1640086654 >> v1:172.21.15.151:6789/0 conn(0x7f9bec14e100 legacy=0x7f9bec064870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:48.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.557+0000 7f9bf24bc700 1 -- 172.21.15.88:0/1640086654 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bec14eb50 con 0x7f9bec14b480 2023-10-18T16:07:48.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.557+0000 7f9bf2cbd700 1 --2- 172.21.15.88:0/1640086654 >> v2:172.21.15.151:3300/0 conn(0x7f9bec064f80 0x7f9bec063b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:07:48.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.558+0000 7f9bf24bc700 1 --2- 172.21.15.88:0/1640086654 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9bec14b480 0x7f9bec0636a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f9bec0c2f00 tx=0x7f9be8009840).ready entity=mon.0 client_cookie=a784feb37aca3369 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:48.566 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.558+0000 7f9bf14ba700 1 -- 172.21.15.88:0/1640086654 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be800e070 con 0x7f9bec14b480 2023-10-18T16:07:48.566 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.558+0000 7f9bf14ba700 1 -- 172.21.15.88:0/1640086654 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9be800ba50 con 0x7f9bec14b480 2023-10-18T16:07:48.566 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.558+0000 7f9bf14ba700 1 -- 172.21.15.88:0/1640086654 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be800bdd0 con 0x7f9bec14b480 2023-10-18T16:07:48.566 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.558+0000 7f9bf4720700 1 -- 172.21.15.88:0/1640086654 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9bec14b480 msgr2=0x7f9bec0636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:48.566 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.558+0000 7f9bf4720700 1 --2- 172.21.15.88:0/1640086654 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9bec14b480 0x7f9bec0636a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f9bec0c2f00 tx=0x7f9be8009840).stop 2023-10-18T16:07:48.567 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.558+0000 7f9bf4720700 1 -- 172.21.15.88:0/1640086654 shutdown_connections 2023-10-18T16:07:48.567 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.558+0000 7f9bf4720700 1 --2- 172.21.15.88:0/1640086654 >> v2:172.21.15.151:3300/0 conn(0x7f9bec064f80 0x7f9bec063b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:48.567 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.558+0000 7f9bf4720700 1 --2- 172.21.15.88:0/1640086654 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9bec14b480 0x7f9bec0636a0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:48.567 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.558+0000 7f9bf4720700 1 -- 172.21.15.88:0/1640086654 >> 172.21.15.88:0/1640086654 conn(0x7f9bec069360 msgr2=0x7f9bec069770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:48.567 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.558+0000 7f9bf4720700 1 -- 172.21.15.88:0/1640086654 shutdown_connections 2023-10-18T16:07:48.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.558+0000 7f9bf4720700 1 -- 172.21.15.88:0/1640086654 wait complete. 2023-10-18T16:07:48.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.559+0000 7f9bf4720700 1 Processor -- start 2023-10-18T16:07:48.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.559+0000 7f9bf4720700 1 -- start start 2023-10-18T16:07:48.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.560+0000 7f9bf4720700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9bec064f80 0x7f9bec0c2820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:48.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.560+0000 7f9bf4720700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9bec14b480 0x7f9bec141910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:48.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.560+0000 7f9bf4720700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9bec15e540 0x7f9bec0c1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:48.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.560+0000 7f9bf4720700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bec162370 con 0x7f9bec14b480 2023-10-18T16:07:48.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.560+0000 7f9bf4720700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bec161ac0 con 0x7f9bec064f80 2023-10-18T16:07:48.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.560+0000 7f9bf4720700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bec142160 con 0x7f9bec15e540 2023-10-18T16:07:48.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.560+0000 7f9bf1cbb700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9bec14b480 0x7f9bec141910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:48.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.560+0000 7f9bf1cbb700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9bec14b480 0x7f9bec141910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59480/0 (socket says 172.21.15.88:59480) 2023-10-18T16:07:48.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.560+0000 7f9bf1cbb700 1 -- 172.21.15.88:0/2292204206 learned_addr learned my addr 172.21.15.88:0/2292204206 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:48.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.560+0000 7f9bf24bc700 1 --2- 172.21.15.88:0/2292204206 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9bec064f80 0x7f9bec0c2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:48.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.560+0000 7f9bf2cbd700 1 --2- 172.21.15.88:0/2292204206 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9bec15e540 0x7f9bec0c1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:48.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.560+0000 7f9bf1cbb700 1 -- 172.21.15.88:0/2292204206 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9bec15e540 msgr2=0x7f9bec0c1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:48.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.560+0000 7f9bf1cbb700 1 --2- 172.21.15.88:0/2292204206 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9bec15e540 0x7f9bec0c1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:48.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.560+0000 7f9bf1cbb700 1 -- 172.21.15.88:0/2292204206 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9bec064f80 msgr2=0x7f9bec0c2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:48.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.560+0000 7f9bf1cbb700 1 --2- 172.21.15.88:0/2292204206 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9bec064f80 0x7f9bec0c2820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:48.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.560+0000 7f9bf1cbb700 1 -- 172.21.15.88:0/2292204206 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9be8009ae0 con 0x7f9bec14b480 2023-10-18T16:07:48.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.561+0000 7f9bf1cbb700 1 --2- 172.21.15.88:0/2292204206 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9bec14b480 0x7f9bec141910 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f9bdc00c440 tx=0x7f9bdc007e00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:48.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.561+0000 7f9be27fc700 1 -- 172.21.15.88:0/2292204206 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bdc007020 con 0x7f9bec14b480 2023-10-18T16:07:48.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.561+0000 7f9be27fc700 1 -- 172.21.15.88:0/2292204206 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bdc007180 con 0x7f9bec14b480 2023-10-18T16:07:48.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.561+0000 7f9bf4720700 1 -- 172.21.15.88:0/2292204206 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bec142440 con 0x7f9bec14b480 2023-10-18T16:07:48.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.561+0000 7f9be27fc700 1 -- 172.21.15.88:0/2292204206 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bdc007520 con 0x7f9bec14b480 2023-10-18T16:07:48.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.561+0000 7f9bf4720700 1 -- 172.21.15.88:0/2292204206 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bec162000 con 0x7f9bec14b480 2023-10-18T16:07:48.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.562+0000 7f9bf4720700 1 -- 172.21.15.88:0/2292204206 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bec162520 con 0x7f9bec14b480 2023-10-18T16:07:48.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.562+0000 7f9be27fc700 1 -- 172.21.15.88:0/2292204206 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f9bdc0068b0 con 0x7f9bec14b480 2023-10-18T16:07:48.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.562+0000 7f9be27fc700 1 --2- 172.21.15.88:0/2292204206 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f9bd005f180 0x7f9bd0061640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:48.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.562+0000 7f9be27fc700 1 -- 172.21.15.88:0/2292204206 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6121+0+0 (secure 0 0 0) 0x7f9bdc079500 con 0x7f9bec14b480 2023-10-18T16:07:48.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.563+0000 7f9bf24bc700 1 --2- 172.21.15.88:0/2292204206 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f9bd005f180 0x7f9bd0061640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:48.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.563+0000 7f9bf24bc700 1 --2- 172.21.15.88:0/2292204206 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f9bd005f180 0x7f9bd0061640 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f9be8009ab0 tx=0x7f9be8009840).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:48.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.566+0000 7f9be27fc700 1 -- 172.21.15.88:0/2292204206 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f9bdc047ac0 con 0x7f9bec14b480 2023-10-18T16:07:48.736 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.732+0000 7f9bf4720700 1 -- 172.21.15.88:0/2292204206 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9bec062900 con 0x7f9bec14b480 2023-10-18T16:07:48.736 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.732+0000 7f9be27fc700 1 -- 172.21.15.88:0/2292204206 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16928 (secure 0 0 0) 0x7f9bec162520 con 0x7f9bec14b480 2023-10-18T16:07:48.737 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:07:48.737 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":22,"fsid":"fb1f77e8-ead0-4f90-8128-fdf5137eb16f","created":"2023-10-18T16:07:23.662468+0000","modified":"2023-10-18T16:07:42.784120+0000","last_up_change":"2023-10-18T16:07:32.498203+0000","last_in_change":"2023-10-18T16:07:26.064784+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-10-18T16:07:32.536320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-18T16:07:34.769989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-10-18T16:07:39.439702+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-10-18T16:07:40.753774+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e421c20c-f7bc-4574-97a2-201bb4a16b4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":91007},{"type":"v1","addr":"172.21.15.88:6809","nonce":91007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":91007},{"type":"v1","addr":"172.21.15.88:6811","nonce":91007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":91007},{"type":"v1","addr":"172.21.15.88:6815","nonce":91007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":91007},{"type":"v1","addr":"172.21.15.88:6813","nonce":91007}]},"public_addr":"172.21.15.88:6809/91007","cluster_addr":"172.21.15.88:6811/91007","heartbeat_back_addr":"172.21.15.88:6815/91007","heartbeat_front_addr":"172.21.15.88:6813/91007","state":["exists","up"]},{"osd":1,"uuid":"9485c5ec-379a-4038-a2d8-1f1081fe2eea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":91009},{"type":"v1","addr":"172.21.15.88:6825","nonce":91009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":91009},{"type":"v1","addr":"172.21.15.88:6827","nonce":91009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":91009},{"type":"v1","addr":"172.21.15.88:6831","nonce":91009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":91009},{"type":"v1","addr":"172.21.15.88:6829","nonce":91009}]},"public_addr":"172.21.15.88:6825/91009","cluster_addr":"172.21.15.88:6827/91009","heartbeat_back_addr":"172.21.15.88:6831/91009","heartbeat_front_addr":"172.21.15.88:6829/91009","state":["exists","up"]},{"osd":2,"uuid":"671895e2-57f3-4f1a-ba4a-a580af1f3ec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":91005},{"type":"v1","addr":"172.21.15.88:6801","nonce":91005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":91005},{"type":"v1","addr":"172.21.15.88:6803","nonce":91005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":91005},{"type":"v1","addr":"172.21.15.88:6807","nonce":91005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":91005},{"type":"v1","addr":"172.21.15.88:6805","nonce":91005}]},"public_addr":"172.21.15.88:6801/91005","cluster_addr":"172.21.15.88:6803/91005","heartbeat_back_addr":"172.21.15.88:6807/91005","heartbeat_front_addr":"172.21.15.88:6805/91005","state":["exists","up"]},{"osd":3,"uuid":"0c4e6e7f-7d9c-4e65-beec-fd0ca384f320","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":91008},{"type":"v1","addr":"172.21.15.88:6817","nonce":91008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":91008},{"type":"v1","addr":"172.21.15.88:6819","nonce":91008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":91008},{"type":"v1","addr":"172.21.15.88:6823","nonce":91008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":91008},{"type":"v1","addr":"172.21.15.88:6821","nonce":91008}]},"public_addr":"172.21.15.88:6817/91008","cluster_addr":"172.21.15.88:6819/91008","heartbeat_back_addr":"172.21.15.88:6823/91008","heartbeat_front_addr":"172.21.15.88:6821/91008","state":["exists","up"]},{"osd":4,"uuid":"29570691-ef35-4b80-9247-aa3703877af4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":91514},{"type":"v1","addr":"172.21.15.151:6801","nonce":91514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":91514},{"type":"v1","addr":"172.21.15.151:6803","nonce":91514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":91514},{"type":"v1","addr":"172.21.15.151:6807","nonce":91514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":91514},{"type":"v1","addr":"172.21.15.151:6805","nonce":91514}]},"public_addr":"172.21.15.151:6801/91514","cluster_addr":"172.21.15.151:6803/91514","heartbeat_back_addr":"172.21.15.151:6807/91514","heartbeat_front_addr":"172.21.15.151:6805/91514","state":["exists","up"]},{"osd":5,"uuid":"26540b7b-032a-4b3b-bf6c-b131a449fa99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":91513},{"type":"v1","addr":"172.21.15.151:6809","nonce":91513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":91513},{"type":"v1","addr":"172.21.15.151:6811","nonce":91513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":91513},{"type":"v1","addr":"172.21.15.151:6815","nonce":91513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":91513},{"type":"v1","addr":"172.21.15.151:6813","nonce":91513}]},"public_addr":"172.21.15.151:6809/91513","cluster_addr":"172.21.15.151:6811/91513","heartbeat_back_addr":"172.21.15.151:6815/91513","heartbeat_front_addr":"172.21.15.151:6813/91513","state":["exists","up"]},{"osd":6,"uuid":"7af01f9c-90db-45fd-93c8-09517c9b0881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":91544},{"type":"v1","addr":"172.21.15.151:6817","nonce":91544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":91544},{"type":"v1","addr":"172.21.15.151:6819","nonce":91544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":91544},{"type":"v1","addr":"172.21.15.151:6823","nonce":91544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":91544},{"type":"v1","addr":"172.21.15.151:6821","nonce":91544}]},"public_addr":"172.21.15.151:6817/91544","cluster_addr":"172.21.15.151:6819/91544","heartbeat_back_addr":"172.21.15.151:6823/91544","heartbeat_front_addr":"172.21.15.151:6821/91544","state":["exists","up"]},{"osd":7,"uuid":"b527b0a5-4249-4854-a30f-bff897757239","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":91545},{"type":"v1","addr":"172.21.15.151:6825","nonce":91545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":91545},{"type":"v1","addr":"172.21.15.151:6827","nonce":91545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":91545},{"type":"v1","addr":"172.21.15.151:6831","nonce":91545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":91545},{"type":"v1","addr":"172.21.15.151:6829","nonce":91545}]},"public_addr":"172.21.15.151:6825/91545","cluster_addr":"172.21.15.151:6827/91545","heartbeat_back_addr":"172.21.15.151:6831/91545","heartbeat_front_addr":"172.21.15.151:6829/91545","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T16:07:31.216742+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T16:07:31.467229+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T16:07:31.220809+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T16:07:31.414999+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T16:07:31.188976+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T16:07:31.252153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T16:07:31.254324+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T16:07:31.203434+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-18T16:07:48.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.734+0000 7f9bf4720700 1 -- 172.21.15.88:0/2292204206 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f9bd005f180 msgr2=0x7f9bd0061640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:48.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.734+0000 7f9bf4720700 1 --2- 172.21.15.88:0/2292204206 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f9bd005f180 0x7f9bd0061640 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f9be8009ab0 tx=0x7f9be8009840).stop 2023-10-18T16:07:48.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.734+0000 7f9bf4720700 1 -- 172.21.15.88:0/2292204206 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9bec14b480 msgr2=0x7f9bec141910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:48.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.734+0000 7f9bf4720700 1 --2- 172.21.15.88:0/2292204206 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9bec14b480 0x7f9bec141910 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f9bdc00c440 tx=0x7f9bdc007e00).stop 2023-10-18T16:07:48.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.734+0000 7f9bf4720700 1 -- 172.21.15.88:0/2292204206 shutdown_connections 2023-10-18T16:07:48.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.734+0000 7f9bf4720700 1 --2- 172.21.15.88:0/2292204206 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f9bd005f180 0x7f9bd0061640 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:48.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.734+0000 7f9bf4720700 1 --2- 172.21.15.88:0/2292204206 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9bec15e540 0x7f9bec0c1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:48.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.734+0000 7f9bf4720700 1 --2- 172.21.15.88:0/2292204206 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9bec064f80 0x7f9bec0c2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:48.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.734+0000 7f9bf4720700 1 --2- 172.21.15.88:0/2292204206 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9bec14b480 0x7f9bec141910 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:48.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.734+0000 7f9bf4720700 1 -- 172.21.15.88:0/2292204206 >> 172.21.15.88:0/2292204206 conn(0x7f9bec069360 msgr2=0x7f9bec145810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:48.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.734+0000 7f9bf4720700 1 -- 172.21.15.88:0/2292204206 shutdown_connections 2023-10-18T16:07:48.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.734+0000 7f9bf4720700 1 -- 172.21.15.88:0/2292204206 wait complete. 2023-10-18T16:07:48.750 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-10-18T16:07:48.864 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.860+0000 7fbff1569700 1 Processor -- start 2023-10-18T16:07:48.864 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.860+0000 7fbff1569700 1 -- start start 2023-10-18T16:07:48.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.861+0000 7fbff1569700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbfec14b530 0x7fbfec14e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:48.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.861+0000 7fbff1569700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbfec067110 0x7fbfec067580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:48.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.861+0000 7fbff1569700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfec0691f0 con 0x7fbfec14b150 2023-10-18T16:07:48.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.861+0000 7fbff1569700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfec067a70 con 0x7fbfec14b530 2023-10-18T16:07:48.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.861+0000 7fbff1569700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfec0685e0 con 0x7fbfec067110 2023-10-18T16:07:48.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.861+0000 7fbfeb7fe700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbfec14b530 0x7fbfec14e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:48.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.861+0000 7fbfeb7fe700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbfec14b530 0x7fbfec14e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59482/0 (socket says 172.21.15.88:59482) 2023-10-18T16:07:48.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.861+0000 7fbfeb7fe700 1 -- 172.21.15.88:0/1834373835 learned_addr learned my addr 172.21.15.88:0/1834373835 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:48.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.861+0000 7fbff0d68700 1 --2- 172.21.15.88:0/1834373835 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbfec067110 0x7fbfec067580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:48.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.862+0000 7fbfeb7fe700 1 -- 172.21.15.88:0/1834373835 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbfec067110 msgr2=0x7fbfec067580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:48.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.862+0000 7fbfeb7fe700 1 --2- 172.21.15.88:0/1834373835 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbfec067110 0x7fbfec067580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:48.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.862+0000 7fbfeb7fe700 1 -- 172.21.15.88:0/1834373835 >> v1:172.21.15.151:6789/0 conn(0x7fbfec14b150 legacy=0x7fbfec0b9670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:48.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.862+0000 7fbfeb7fe700 1 -- 172.21.15.88:0/1834373835 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfec0688c0 con 0x7fbfec14b530 2023-10-18T16:07:48.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.862+0000 7fbff0d68700 1 --2- 172.21.15.88:0/1834373835 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbfec067110 0x7fbfec067580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-10-18T16:07:48.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.862+0000 7fbfeb7fe700 1 --2- 172.21.15.88:0/1834373835 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbfec14b530 0x7fbfec14e810 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fbfe0006fd0 tx=0x7fbfe00090f0).ready entity=mon.0 client_cookie=fe39ad7180710e37 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:48.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.862+0000 7fbfeaffd700 1 -- 172.21.15.88:0/1834373835 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfe000b600 con 0x7fbfec14b530 2023-10-18T16:07:48.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.862+0000 7fbfeaffd700 1 -- 172.21.15.88:0/1834373835 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfe000b760 con 0x7fbfec14b530 2023-10-18T16:07:48.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.862+0000 7fbfeaffd700 1 -- 172.21.15.88:0/1834373835 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfe000bae0 con 0x7fbfec14b530 2023-10-18T16:07:48.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.862+0000 7fbff1569700 1 -- 172.21.15.88:0/1834373835 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbfec14b530 msgr2=0x7fbfec14e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:48.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.862+0000 7fbff1569700 1 --2- 172.21.15.88:0/1834373835 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbfec14b530 0x7fbfec14e810 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fbfe0006fd0 tx=0x7fbfe00090f0).stop 2023-10-18T16:07:48.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.862+0000 7fbff1569700 1 -- 172.21.15.88:0/1834373835 shutdown_connections 2023-10-18T16:07:48.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.862+0000 7fbff1569700 1 --2- 172.21.15.88:0/1834373835 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbfec067110 0x7fbfec067580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:48.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.862+0000 7fbff1569700 1 --2- 172.21.15.88:0/1834373835 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbfec14b530 0x7fbfec14e810 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:48.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.862+0000 7fbff1569700 1 -- 172.21.15.88:0/1834373835 >> 172.21.15.88:0/1834373835 conn(0x7fbfec062650 msgr2=0x7fbfec062a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:48.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.862+0000 7fbff1569700 1 -- 172.21.15.88:0/1834373835 shutdown_connections 2023-10-18T16:07:48.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.862+0000 7fbff1569700 1 -- 172.21.15.88:0/1834373835 wait complete. 2023-10-18T16:07:48.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.863+0000 7fbff1569700 1 Processor -- start 2023-10-18T16:07:48.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.864+0000 7fbff1569700 1 -- start start 2023-10-18T16:07:48.875 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.864+0000 7fbff1569700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbfec067110 0x7fbfec0c2480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:48.875 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.864+0000 7fbff1569700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbfec14b150 0x7fbfec14e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:48.876 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.864+0000 7fbff1569700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbfec0c0b60 0x7fbfec15e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:48.876 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.864+0000 7fbff1569700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfec15dd60 con 0x7fbfec14b150 2023-10-18T16:07:48.876 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.864+0000 7fbff1569700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfec0c2c50 con 0x7fbfec067110 2023-10-18T16:07:48.876 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.864+0000 7fbff1569700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfec0c43b0 con 0x7fbfec0c0b60 2023-10-18T16:07:48.877 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.864+0000 7fbfeb7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbfec14b150 0x7fbfec14e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:48.877 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.864+0000 7fbfeb7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbfec14b150 0x7fbfec14e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59494/0 (socket says 172.21.15.88:59494) 2023-10-18T16:07:48.877 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.864+0000 7fbfeb7fe700 1 -- 172.21.15.88:0/664979438 learned_addr learned my addr 172.21.15.88:0/664979438 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:48.877 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.865+0000 7fbff0d68700 1 --2- 172.21.15.88:0/664979438 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbfec0c0b60 0x7fbfec15e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:48.877 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.865+0000 7fbfebfff700 1 --2- 172.21.15.88:0/664979438 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbfec067110 0x7fbfec0c2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:48.877 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.865+0000 7fbfeb7fe700 1 -- 172.21.15.88:0/664979438 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbfec0c0b60 msgr2=0x7fbfec15e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:48.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.865+0000 7fbfeb7fe700 1 --2- 172.21.15.88:0/664979438 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbfec0c0b60 0x7fbfec15e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:48.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.865+0000 7fbfeb7fe700 1 -- 172.21.15.88:0/664979438 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbfec067110 msgr2=0x7fbfec0c2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:48.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.865+0000 7fbfeb7fe700 1 --2- 172.21.15.88:0/664979438 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbfec067110 0x7fbfec0c2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:48.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.865+0000 7fbfeb7fe700 1 -- 172.21.15.88:0/664979438 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfe0009370 con 0x7fbfec14b150 2023-10-18T16:07:48.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.865+0000 7fbfebfff700 1 --2- 172.21.15.88:0/664979438 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbfec067110 0x7fbfec0c2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:07:48.879 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.865+0000 7fbfeb7fe700 1 --2- 172.21.15.88:0/664979438 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbfec14b150 0x7fbfec14e4d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fbfec0695b0 tx=0x7fbfe0017000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:48.879 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.865+0000 7fbfd3fff700 1 -- 172.21.15.88:0/664979438 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfe000b8d0 con 0x7fbfec14b150 2023-10-18T16:07:48.879 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.865+0000 7fbfd3fff700 1 -- 172.21.15.88:0/664979438 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfe000ba30 con 0x7fbfec14b150 2023-10-18T16:07:48.879 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.865+0000 7fbff1569700 1 -- 172.21.15.88:0/664979438 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfec0c4690 con 0x7fbfec14b150 2023-10-18T16:07:48.879 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.865+0000 7fbfd3fff700 1 -- 172.21.15.88:0/664979438 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfe001b660 con 0x7fbfec14b150 2023-10-18T16:07:48.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.865+0000 7fbff1569700 1 -- 172.21.15.88:0/664979438 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfec0c4c50 con 0x7fbfec14b150 2023-10-18T16:07:48.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.866+0000 7fbff1569700 1 -- 172.21.15.88:0/664979438 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fbfec069fd0 con 0x7fbfec14b150 2023-10-18T16:07:48.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.866+0000 7fbfd3fff700 1 -- 172.21.15.88:0/664979438 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7fbfe001ba10 con 0x7fbfec14b150 2023-10-18T16:07:48.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.866+0000 7fbfd3fff700 1 --2- 172.21.15.88:0/664979438 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fbfd805f150 0x7fbfd8061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:48.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.867+0000 7fbfd3fff700 1 -- 172.21.15.88:0/664979438 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6121+0+0 (secure 0 0 0) 0x7fbfe007ba90 con 0x7fbfec14b150 2023-10-18T16:07:48.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.867+0000 7fbfd3fff700 1 --2- 172.21.15.88:0/664979438 >> [v2:172.21.15.88:6808/91007,v1:172.21.15.88:6809/91007] conn(0x7fbfd80684f0 0x7fbfd806a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:48.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.867+0000 7fbfd3fff700 1 -- 172.21.15.88:0/664979438 --> [v2:172.21.15.88:6808/91007,v1:172.21.15.88:6809/91007] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbfd806afd0 con 0x7fbfd80684f0 2023-10-18T16:07:48.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.867+0000 7fbfebfff700 1 --2- 172.21.15.88:0/664979438 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fbfd805f150 0x7fbfd8061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:48.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.867+0000 7fbfd3fff700 1 -- 172.21.15.88:0/664979438 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbfe0080220 con 0x7fbfec14b150 2023-10-18T16:07:48.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.867+0000 7fbff0d68700 1 --2- 172.21.15.88:0/664979438 >> [v2:172.21.15.88:6808/91007,v1:172.21.15.88:6809/91007] conn(0x7fbfd80684f0 0x7fbfd806a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:48.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.867+0000 7fbfebfff700 1 --2- 172.21.15.88:0/664979438 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fbfd805f150 0x7fbfd8061610 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7fbfdc00b8b0 tx=0x7fbfdc008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:48.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.867+0000 7fbff0d68700 1 --2- 172.21.15.88:0/664979438 >> [v2:172.21.15.88:6808/91007,v1:172.21.15.88:6809/91007] conn(0x7fbfd80684f0 0x7fbfd806a950 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:48.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.868+0000 7fbfd3fff700 1 -- 172.21.15.88:0/664979438 <== osd.0 v2:172.21.15.88:6808/91007 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23065 (crc 0 0 0) 0x7fbfd806afd0 con 0x7fbfd80684f0 2023-10-18T16:07:48.885 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.882+0000 7fbff1569700 1 -- 172.21.15.88:0/664979438 --> [v2:172.21.15.88:6808/91007,v1:172.21.15.88:6809/91007] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fbfec15deb0 con 0x7fbfd80684f0 2023-10-18T16:07:48.886 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.882+0000 7fbfd3fff700 1 -- 172.21.15.88:0/664979438 <== osd.0 v2:172.21.15.88:6808/91007 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fbfec15deb0 con 0x7fbfd80684f0 2023-10-18T16:07:48.887 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.882+0000 7fbff1569700 1 -- 172.21.15.88:0/664979438 >> [v2:172.21.15.88:6808/91007,v1:172.21.15.88:6809/91007] conn(0x7fbfd80684f0 msgr2=0x7fbfd806a950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:48.887 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.882+0000 7fbff1569700 1 --2- 172.21.15.88:0/664979438 >> [v2:172.21.15.88:6808/91007,v1:172.21.15.88:6809/91007] conn(0x7fbfd80684f0 0x7fbfd806a950 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:48.887 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.883+0000 7fbff1569700 1 -- 172.21.15.88:0/664979438 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fbfd805f150 msgr2=0x7fbfd8061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:48.887 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.883+0000 7fbff1569700 1 --2- 172.21.15.88:0/664979438 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fbfd805f150 0x7fbfd8061610 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7fbfdc00b8b0 tx=0x7fbfdc008040).stop 2023-10-18T16:07:48.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.883+0000 7fbff1569700 1 -- 172.21.15.88:0/664979438 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbfec14b150 msgr2=0x7fbfec14e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:48.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.883+0000 7fbff1569700 1 --2- 172.21.15.88:0/664979438 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbfec14b150 0x7fbfec14e4d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fbfec0695b0 tx=0x7fbfe0017000).stop 2023-10-18T16:07:48.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.883+0000 7fbff0d68700 1 -- 172.21.15.88:0/664979438 reap_dead start 2023-10-18T16:07:48.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.883+0000 7fbff1569700 1 -- 172.21.15.88:0/664979438 shutdown_connections 2023-10-18T16:07:48.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.883+0000 7fbff1569700 1 -- 172.21.15.88:0/664979438 >> 172.21.15.88:0/664979438 conn(0x7fbfec062650 msgr2=0x7fbfec14fe80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:48.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.883+0000 7fbff1569700 1 -- 172.21.15.88:0/664979438 shutdown_connections 2023-10-18T16:07:48.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:48.883+0000 7fbff1569700 1 -- 172.21.15.88:0/664979438 wait complete. 2023-10-18T16:07:48.897 INFO:teuthology.orchestra.run.smithi088.stdout:55834574853 2023-10-18T16:07:48.898 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-10-18T16:07:49.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.008+0000 7f3780f7d700 1 Processor -- start 2023-10-18T16:07:49.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.008+0000 7f3780f7d700 1 -- start start 2023-10-18T16:07:49.014 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.009+0000 7f3780f7d700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f377c0c20a0 0x7f377c154280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:49.014 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.009+0000 7f3780f7d700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f377c150340 0x7f377c1507f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:49.015 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.009+0000 7f3780f7d700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f377c0a8570 con 0x7f377c14d880 2023-10-18T16:07:49.015 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.009+0000 7f3780f7d700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f377c154770 con 0x7f377c150340 2023-10-18T16:07:49.015 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.009+0000 7f3780f7d700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f377c0a87c0 con 0x7f377c0c20a0 2023-10-18T16:07:49.015 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.009+0000 7f377ad9d700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f377c150340 0x7f377c1507f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:49.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.009+0000 7f3779d9b700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f377c0c20a0 0x7f377c154280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:49.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.009+0000 7f377a59c700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f377c14d880 0x7f377c0c1990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.88:52570/0 (socket says 172.21.15.88:52570) 2023-10-18T16:07:49.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.009+0000 7f377a59c700 1 -- 172.21.15.88:0/2685131460 learned_addr learned my addr 172.21.15.88:0/2685131460 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:07:49.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.010+0000 7f3779d9b700 1 -- 172.21.15.88:0/2685131460 >> v2:172.21.15.151:3300/0 conn(0x7f377c150340 msgr2=0x7f377c1507f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:49.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.010+0000 7f3779d9b700 1 --2- 172.21.15.88:0/2685131460 >> v2:172.21.15.151:3300/0 conn(0x7f377c150340 0x7f377c1507f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:49.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.010+0000 7f3779d9b700 1 -- 172.21.15.88:0/2685131460 >> v1:172.21.15.151:6789/0 conn(0x7f377c14d880 legacy=0x7f377c0c1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:49.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.010+0000 7f3779d9b700 1 -- 172.21.15.88:0/2685131460 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f377c0a8aa0 con 0x7f377c0c20a0 2023-10-18T16:07:49.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.010+0000 7f3779d9b700 1 --2- 172.21.15.88:0/2685131460 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f377c0c20a0 0x7f377c154280 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f3770009760 tx=0x7f3770009060).ready entity=mon.2 client_cookie=2f45f0eef482a5e0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:49.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.010+0000 7f377959a700 1 -- 172.21.15.88:0/2685131460 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f377000f510 con 0x7f377c0c20a0 2023-10-18T16:07:49.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.010+0000 7f377959a700 1 -- 172.21.15.88:0/2685131460 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f377000f670 con 0x7f377c0c20a0 2023-10-18T16:07:49.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.010+0000 7f377959a700 1 -- 172.21.15.88:0/2685131460 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f377000f9f0 con 0x7f377c0c20a0 2023-10-18T16:07:49.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.010+0000 7f3780f7d700 1 -- 172.21.15.88:0/2685131460 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f377c0c20a0 msgr2=0x7f377c154280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:49.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.010+0000 7f3780f7d700 1 --2- 172.21.15.88:0/2685131460 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f377c0c20a0 0x7f377c154280 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f3770009760 tx=0x7f3770009060).stop 2023-10-18T16:07:49.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.011+0000 7f3780f7d700 1 -- 172.21.15.88:0/2685131460 shutdown_connections 2023-10-18T16:07:49.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.011+0000 7f3780f7d700 1 --2- 172.21.15.88:0/2685131460 >> v2:172.21.15.151:3300/0 conn(0x7f377c150340 0x7f377c1507f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:49.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.011+0000 7f3780f7d700 1 --2- 172.21.15.88:0/2685131460 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f377c0c20a0 0x7f377c154280 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:49.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.011+0000 7f3780f7d700 1 -- 172.21.15.88:0/2685131460 >> 172.21.15.88:0/2685131460 conn(0x7f377c069d80 msgr2=0x7f377c06a190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:49.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.011+0000 7f3780f7d700 1 -- 172.21.15.88:0/2685131460 shutdown_connections 2023-10-18T16:07:49.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.011+0000 7f3780f7d700 1 -- 172.21.15.88:0/2685131460 wait complete. 2023-10-18T16:07:49.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.012+0000 7f3780f7d700 1 Processor -- start 2023-10-18T16:07:49.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.012+0000 7f3780f7d700 1 -- start start 2023-10-18T16:07:49.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.012+0000 7f3780f7d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f377c0c20a0 0x7f377c14c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:49.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.012+0000 7f3780f7d700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f377c14d880 0x7f377c0ba1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:49.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.012+0000 7f3780f7d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f377c150340 0x7f377c0aaaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:49.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.012+0000 7f3780f7d700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f377c0beee0 con 0x7f377c150340 2023-10-18T16:07:49.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.012+0000 7f3780f7d700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f377c0bfdf0 con 0x7f377c0c20a0 2023-10-18T16:07:49.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.012+0000 7f3780f7d700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f377c0c00d0 con 0x7f377c14d880 2023-10-18T16:07:49.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.012+0000 7f377ad9d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f377c150340 0x7f377c0aaaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:49.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.012+0000 7f377ad9d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f377c150340 0x7f377c0aaaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59508/0 (socket says 172.21.15.88:59508) 2023-10-18T16:07:49.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.012+0000 7f377ad9d700 1 -- 172.21.15.88:0/1508615871 learned_addr learned my addr 172.21.15.88:0/1508615871 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:49.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.012+0000 7f3779d9b700 1 --2- 172.21.15.88:0/1508615871 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f377c14d880 0x7f377c0ba1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:49.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.012+0000 7f377a59c700 1 --2- 172.21.15.88:0/1508615871 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f377c0c20a0 0x7f377c14c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:49.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.013+0000 7f377ad9d700 1 -- 172.21.15.88:0/1508615871 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f377c14d880 msgr2=0x7f377c0ba1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:49.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.013+0000 7f377ad9d700 1 --2- 172.21.15.88:0/1508615871 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f377c14d880 0x7f377c0ba1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:49.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.013+0000 7f377ad9d700 1 -- 172.21.15.88:0/1508615871 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f377c0c20a0 msgr2=0x7f377c14c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:49.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.013+0000 7f377ad9d700 1 --2- 172.21.15.88:0/1508615871 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f377c0c20a0 0x7f377c14c0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:49.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.013+0000 7f377ad9d700 1 -- 172.21.15.88:0/1508615871 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37700092a0 con 0x7f377c150340 2023-10-18T16:07:49.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.013+0000 7f377ad9d700 1 --2- 172.21.15.88:0/1508615871 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f377c150340 0x7f377c0aaaa0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f377400e900 tx=0x7f3774009060).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:49.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.013+0000 7f376a7fc700 1 -- 172.21.15.88:0/1508615871 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3774012600 con 0x7f377c150340 2023-10-18T16:07:49.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.013+0000 7f376a7fc700 1 -- 172.21.15.88:0/1508615871 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3774012760 con 0x7f377c150340 2023-10-18T16:07:49.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.013+0000 7f3780f7d700 1 -- 172.21.15.88:0/1508615871 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f377c0c03b0 con 0x7f377c150340 2023-10-18T16:07:49.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.013+0000 7f376a7fc700 1 -- 172.21.15.88:0/1508615871 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3774012ae0 con 0x7f377c150340 2023-10-18T16:07:49.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.013+0000 7f3780f7d700 1 -- 172.21.15.88:0/1508615871 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f377c0c0970 con 0x7f377c150340 2023-10-18T16:07:49.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.014+0000 7f3780f7d700 1 -- 172.21.15.88:0/1508615871 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f377c06c4e0 con 0x7f377c150340 2023-10-18T16:07:49.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.014+0000 7f376a7fc700 1 -- 172.21.15.88:0/1508615871 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f377401b400 con 0x7f377c150340 2023-10-18T16:07:49.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.014+0000 7f376a7fc700 1 --2- 172.21.15.88:0/1508615871 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f375c05f150 0x7f375c061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:49.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.015+0000 7f376a7fc700 1 -- 172.21.15.88:0/1508615871 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6121+0+0 (secure 0 0 0) 0x7f377407a500 con 0x7f377c150340 2023-10-18T16:07:49.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.015+0000 7f377a59c700 1 --2- 172.21.15.88:0/1508615871 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f375c05f150 0x7f375c061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:49.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.015+0000 7f376a7fc700 1 --2- 172.21.15.88:0/1508615871 >> [v2:172.21.15.88:6824/91009,v1:172.21.15.88:6825/91009] conn(0x7f375c0684f0 0x7f375c06a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:49.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.015+0000 7f376a7fc700 1 -- 172.21.15.88:0/1508615871 --> [v2:172.21.15.88:6824/91009,v1:172.21.15.88:6825/91009] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f375c06afd0 con 0x7f375c0684f0 2023-10-18T16:07:49.029 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.015+0000 7f376a7fc700 1 -- 172.21.15.88:0/1508615871 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f377407cb20 con 0x7f377c150340 2023-10-18T16:07:49.029 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.015+0000 7f3779d9b700 1 --2- 172.21.15.88:0/1508615871 >> [v2:172.21.15.88:6824/91009,v1:172.21.15.88:6825/91009] conn(0x7f375c0684f0 0x7f375c06a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:49.029 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.015+0000 7f377a59c700 1 --2- 172.21.15.88:0/1508615871 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f375c05f150 0x7f375c061610 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f376c001a90 tx=0x7f376c001560).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:49.029 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.015+0000 7f3779d9b700 1 --2- 172.21.15.88:0/1508615871 >> [v2:172.21.15.88:6824/91009,v1:172.21.15.88:6825/91009] conn(0x7f375c0684f0 0x7f375c06a950 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:49.029 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.016+0000 7f376a7fc700 1 -- 172.21.15.88:0/1508615871 <== osd.1 v2:172.21.15.88:6824/91009 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23065 (crc 0 0 0) 0x7f375c06afd0 con 0x7f375c0684f0 2023-10-18T16:07:49.033 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.030+0000 7f3780f7d700 1 -- 172.21.15.88:0/1508615871 --> [v2:172.21.15.88:6824/91009,v1:172.21.15.88:6825/91009] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f377c0a8d80 con 0x7f375c0684f0 2023-10-18T16:07:49.033 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.030+0000 7f376a7fc700 1 -- 172.21.15.88:0/1508615871 <== osd.1 v2:172.21.15.88:6824/91009 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f377c0a8d80 con 0x7f375c0684f0 2023-10-18T16:07:49.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.030+0000 7f3780f7d700 1 -- 172.21.15.88:0/1508615871 >> [v2:172.21.15.88:6824/91009,v1:172.21.15.88:6825/91009] conn(0x7f375c0684f0 msgr2=0x7f375c06a950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:49.035 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.030+0000 7f3780f7d700 1 --2- 172.21.15.88:0/1508615871 >> [v2:172.21.15.88:6824/91009,v1:172.21.15.88:6825/91009] conn(0x7f375c0684f0 0x7f375c06a950 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:49.035 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.030+0000 7f3780f7d700 1 -- 172.21.15.88:0/1508615871 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f375c05f150 msgr2=0x7f375c061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:49.035 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.030+0000 7f3780f7d700 1 --2- 172.21.15.88:0/1508615871 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f375c05f150 0x7f375c061610 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f376c001a90 tx=0x7f376c001560).stop 2023-10-18T16:07:49.035 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.030+0000 7f3780f7d700 1 -- 172.21.15.88:0/1508615871 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f377c150340 msgr2=0x7f377c0aaaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:49.036 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.030+0000 7f3780f7d700 1 --2- 172.21.15.88:0/1508615871 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f377c150340 0x7f377c0aaaa0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f377400e900 tx=0x7f3774009060).stop 2023-10-18T16:07:49.036 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.030+0000 7f377ad9d700 1 -- 172.21.15.88:0/1508615871 reap_dead start 2023-10-18T16:07:49.036 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.030+0000 7f3780f7d700 1 -- 172.21.15.88:0/1508615871 shutdown_connections 2023-10-18T16:07:49.036 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.030+0000 7f3780f7d700 1 -- 172.21.15.88:0/1508615871 >> 172.21.15.88:0/1508615871 conn(0x7f377c069d80 msgr2=0x7f377c14f880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:49.036 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.030+0000 7f3780f7d700 1 -- 172.21.15.88:0/1508615871 shutdown_connections 2023-10-18T16:07:49.037 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.030+0000 7f3780f7d700 1 -- 172.21.15.88:0/1508615871 wait complete. 2023-10-18T16:07:49.045 INFO:teuthology.orchestra.run.smithi088.stdout:55834574853 2023-10-18T16:07:49.046 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-10-18T16:07:49.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.155+0000 7f22c085b700 1 Processor -- start 2023-10-18T16:07:49.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.155+0000 7f22c085b700 1 -- start start 2023-10-18T16:07:49.162 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.156+0000 7f22c085b700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f22bc14d880 0x7f22bc154280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:49.162 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.156+0000 7f22c085b700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f22bc0c1990 0x7f22bc0c1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:49.162 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.156+0000 7f22c085b700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f22bc064490 0x7f22bc064920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:49.162 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.156+0000 7f22c085b700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22bc0a8570 con 0x7f22bc064490 2023-10-18T16:07:49.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.156+0000 7f22c085b700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22bc154770 con 0x7f22bc14d880 2023-10-18T16:07:49.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.156+0000 7f22c085b700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22bc0a87c0 con 0x7f22bc0c1990 2023-10-18T16:07:49.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.156+0000 7f22bad9d700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f22bc064490 0x7f22bc064920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:49.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.156+0000 7f22bad9d700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f22bc064490 0x7f22bc064920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59512/0 (socket says 172.21.15.88:59512) 2023-10-18T16:07:49.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.156+0000 7f22bad9d700 1 -- 172.21.15.88:0/1696980904 learned_addr learned my addr 172.21.15.88:0/1696980904 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:49.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.157+0000 7f22ba59c700 1 --2- 172.21.15.88:0/1696980904 >> v2:172.21.15.151:3300/0 conn(0x7f22bc14d880 0x7f22bc154280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:49.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.157+0000 7f22b9d9b700 1 --2- 172.21.15.88:0/1696980904 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f22bc0c1990 0x7f22bc0c1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:49.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.157+0000 7f22bad9d700 1 -- 172.21.15.88:0/1696980904 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f22bc0c1990 msgr2=0x7f22bc0c1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:49.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.157+0000 7f22bad9d700 1 --2- 172.21.15.88:0/1696980904 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f22bc0c1990 0x7f22bc0c1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:49.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.157+0000 7f22bad9d700 1 -- 172.21.15.88:0/1696980904 >> v2:172.21.15.151:3300/0 conn(0x7f22bc14d880 msgr2=0x7f22bc154280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:49.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.157+0000 7f22bad9d700 1 --2- 172.21.15.88:0/1696980904 >> v2:172.21.15.151:3300/0 conn(0x7f22bc14d880 0x7f22bc154280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:49.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.157+0000 7f22bad9d700 1 -- 172.21.15.88:0/1696980904 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22bc0a8aa0 con 0x7f22bc064490 2023-10-18T16:07:49.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.157+0000 7f22ba59c700 1 --2- 172.21.15.88:0/1696980904 >> v2:172.21.15.151:3300/0 conn(0x7f22bc14d880 0x7f22bc154280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:07:49.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.157+0000 7f22b9d9b700 1 --2- 172.21.15.88:0/1696980904 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f22bc0c1990 0x7f22bc0c1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:07:49.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.157+0000 7f22bad9d700 1 --2- 172.21.15.88:0/1696980904 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f22bc064490 0x7f22bc064920 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f22bc062ae0 tx=0x7f22b40090f0).ready entity=mon.0 client_cookie=f01759b259aa0d82 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:49.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.157+0000 7f22b959a700 1 -- 172.21.15.88:0/1696980904 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22b400b600 con 0x7f22bc064490 2023-10-18T16:07:49.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.157+0000 7f22b959a700 1 -- 172.21.15.88:0/1696980904 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22b400b760 con 0x7f22bc064490 2023-10-18T16:07:49.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.157+0000 7f22b959a700 1 -- 172.21.15.88:0/1696980904 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22b400bae0 con 0x7f22bc064490 2023-10-18T16:07:49.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.157+0000 7f22c085b700 1 -- 172.21.15.88:0/1696980904 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f22bc064490 msgr2=0x7f22bc064920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:49.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.157+0000 7f22c085b700 1 --2- 172.21.15.88:0/1696980904 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f22bc064490 0x7f22bc064920 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f22bc062ae0 tx=0x7f22b40090f0).stop 2023-10-18T16:07:49.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.157+0000 7f22c085b700 1 -- 172.21.15.88:0/1696980904 shutdown_connections 2023-10-18T16:07:49.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.157+0000 7f22c085b700 1 --2- 172.21.15.88:0/1696980904 >> v2:172.21.15.151:3300/0 conn(0x7f22bc14d880 0x7f22bc154280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:49.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.157+0000 7f22c085b700 1 --2- 172.21.15.88:0/1696980904 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f22bc064490 0x7f22bc064920 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:49.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.157+0000 7f22c085b700 1 --2- 172.21.15.88:0/1696980904 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f22bc0c1990 0x7f22bc0c1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:49.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.157+0000 7f22c085b700 1 -- 172.21.15.88:0/1696980904 >> 172.21.15.88:0/1696980904 conn(0x7f22bc069d80 msgr2=0x7f22bc06a190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:49.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.157+0000 7f22c085b700 1 -- 172.21.15.88:0/1696980904 shutdown_connections 2023-10-18T16:07:49.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.158+0000 7f22c085b700 1 -- 172.21.15.88:0/1696980904 wait complete. 2023-10-18T16:07:49.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.158+0000 7f22c085b700 1 Processor -- start 2023-10-18T16:07:49.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.159+0000 7f22c085b700 1 -- start start 2023-10-18T16:07:49.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.159+0000 7f22c085b700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f22bc064490 0x7f22bc0ab4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:49.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.159+0000 7f22c085b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f22bc0c1990 0x7f22bc0bb380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:49.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.159+0000 7f22c085b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f22bc14d880 0x7f22bc14b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:49.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.159+0000 7f22c085b700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22bc1457b0 con 0x7f22bc14d880 2023-10-18T16:07:49.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.159+0000 7f22c085b700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22bc0bba60 con 0x7f22bc0c1990 2023-10-18T16:07:49.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.159+0000 7f22c085b700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22bc0bb870 con 0x7f22bc064490 2023-10-18T16:07:49.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.159+0000 7f22bad9d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f22bc14d880 0x7f22bc14b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:49.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.159+0000 7f22bad9d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f22bc14d880 0x7f22bc14b570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59528/0 (socket says 172.21.15.88:59528) 2023-10-18T16:07:49.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.159+0000 7f22bad9d700 1 -- 172.21.15.88:0/3036714126 learned_addr learned my addr 172.21.15.88:0/3036714126 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:49.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.159+0000 7f22bad9d700 1 -- 172.21.15.88:0/3036714126 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f22bc064490 msgr2=0x7f22bc0ab4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:49.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.159+0000 7f22bad9d700 1 --2- 172.21.15.88:0/3036714126 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f22bc064490 0x7f22bc0ab4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-10-18T16:07:49.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.159+0000 7f22bad9d700 1 -- 172.21.15.88:0/3036714126 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f22bc0c1990 msgr2=0x7f22bc0bb380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:49.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.159+0000 7f22bad9d700 1 --2- 172.21.15.88:0/3036714126 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f22bc0c1990 0x7f22bc0bb380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-10-18T16:07:49.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.160+0000 7f22bad9d700 1 -- 172.21.15.88:0/3036714126 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22b4009370 con 0x7f22bc14d880 2023-10-18T16:07:49.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.160+0000 7f22bad9d700 1 --2- 172.21.15.88:0/3036714126 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f22bc14d880 0x7f22bc14b570 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f22bc084360 tx=0x7f22b4017000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:49.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.160+0000 7f22aa7fc700 1 -- 172.21.15.88:0/3036714126 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22b400bae0 con 0x7f22bc14d880 2023-10-18T16:07:49.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.160+0000 7f22aa7fc700 1 -- 172.21.15.88:0/3036714126 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22b401b410 con 0x7f22bc14d880 2023-10-18T16:07:49.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.160+0000 7f22c085b700 1 -- 172.21.15.88:0/3036714126 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22bc0a9f70 con 0x7f22bc14d880 2023-10-18T16:07:49.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.160+0000 7f22aa7fc700 1 -- 172.21.15.88:0/3036714126 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22b401b7d0 con 0x7f22bc14d880 2023-10-18T16:07:49.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.160+0000 7f22c085b700 1 -- 172.21.15.88:0/3036714126 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22bc14b0b0 con 0x7f22bc14d880 2023-10-18T16:07:49.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.161+0000 7f22c085b700 1 -- 172.21.15.88:0/3036714126 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f22bc06c4e0 con 0x7f22bc14d880 2023-10-18T16:07:49.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.161+0000 7f22aa7fc700 1 -- 172.21.15.88:0/3036714126 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f22b401bc00 con 0x7f22bc14d880 2023-10-18T16:07:49.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.161+0000 7f22aa7fc700 1 --2- 172.21.15.88:0/3036714126 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f229c05f180 0x7f229c061640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:49.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.161+0000 7f22aa7fc700 1 -- 172.21.15.88:0/3036714126 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6121+0+0 (secure 0 0 0) 0x7f22b407c1c0 con 0x7f22bc14d880 2023-10-18T16:07:49.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.161+0000 7f22aa7fc700 1 --2- 172.21.15.88:0/3036714126 >> [v2:172.21.15.88:6800/91005,v1:172.21.15.88:6801/91005] conn(0x7f229c068520 0x7f229c06a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:49.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.161+0000 7f22aa7fc700 1 -- 172.21.15.88:0/3036714126 --> [v2:172.21.15.88:6800/91005,v1:172.21.15.88:6801/91005] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f229c06b000 con 0x7f229c068520 2023-10-18T16:07:49.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.161+0000 7f22ba59c700 1 --2- 172.21.15.88:0/3036714126 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f229c05f180 0x7f229c061640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:49.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.161+0000 7f22aa7fc700 1 -- 172.21.15.88:0/3036714126 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f22b407e7e0 con 0x7f22bc14d880 2023-10-18T16:07:49.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.162+0000 7f22b9d9b700 1 --2- 172.21.15.88:0/3036714126 >> [v2:172.21.15.88:6800/91005,v1:172.21.15.88:6801/91005] conn(0x7f229c068520 0x7f229c06a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:49.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.162+0000 7f22ba59c700 1 --2- 172.21.15.88:0/3036714126 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f229c05f180 0x7f229c061640 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f22ac0094d0 tx=0x7f22ac009590).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:49.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.162+0000 7f22b9d9b700 1 --2- 172.21.15.88:0/3036714126 >> [v2:172.21.15.88:6800/91005,v1:172.21.15.88:6801/91005] conn(0x7f229c068520 0x7f229c06a980 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:49.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.162+0000 7f22aa7fc700 1 -- 172.21.15.88:0/3036714126 <== osd.2 v2:172.21.15.88:6800/91005 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23065 (crc 0 0 0) 0x7f229c06b000 con 0x7f229c068520 2023-10-18T16:07:49.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.176+0000 7f22c085b700 1 -- 172.21.15.88:0/3036714126 --> [v2:172.21.15.88:6800/91005,v1:172.21.15.88:6801/91005] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f22bc0a8d80 con 0x7f229c068520 2023-10-18T16:07:49.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.177+0000 7f22aa7fc700 1 -- 172.21.15.88:0/3036714126 <== osd.2 v2:172.21.15.88:6800/91005 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f22bc0a8d80 con 0x7f229c068520 2023-10-18T16:07:49.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.177+0000 7f22c085b700 1 -- 172.21.15.88:0/3036714126 >> [v2:172.21.15.88:6800/91005,v1:172.21.15.88:6801/91005] conn(0x7f229c068520 msgr2=0x7f229c06a980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:49.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.177+0000 7f22c085b700 1 --2- 172.21.15.88:0/3036714126 >> [v2:172.21.15.88:6800/91005,v1:172.21.15.88:6801/91005] conn(0x7f229c068520 0x7f229c06a980 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:49.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.177+0000 7f22c085b700 1 -- 172.21.15.88:0/3036714126 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f229c05f180 msgr2=0x7f229c061640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:49.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.177+0000 7f22c085b700 1 --2- 172.21.15.88:0/3036714126 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f229c05f180 0x7f229c061640 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f22ac0094d0 tx=0x7f22ac009590).stop 2023-10-18T16:07:49.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.177+0000 7f22c085b700 1 -- 172.21.15.88:0/3036714126 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f22bc14d880 msgr2=0x7f22bc14b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:49.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.177+0000 7f22c085b700 1 --2- 172.21.15.88:0/3036714126 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f22bc14d880 0x7f22bc14b570 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f22bc084360 tx=0x7f22b4017000).stop 2023-10-18T16:07:49.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.177+0000 7f22bad9d700 1 -- 172.21.15.88:0/3036714126 reap_dead start 2023-10-18T16:07:49.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.177+0000 7f22c085b700 1 -- 172.21.15.88:0/3036714126 shutdown_connections 2023-10-18T16:07:49.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.177+0000 7f22c085b700 1 -- 172.21.15.88:0/3036714126 >> 172.21.15.88:0/3036714126 conn(0x7f22bc069d80 msgr2=0x7f22bc153f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:49.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.177+0000 7f22c085b700 1 -- 172.21.15.88:0/3036714126 shutdown_connections 2023-10-18T16:07:49.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.177+0000 7f22c085b700 1 -- 172.21.15.88:0/3036714126 wait complete. 2023-10-18T16:07:49.192 INFO:teuthology.orchestra.run.smithi088.stdout:55834574853 2023-10-18T16:07:49.192 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-10-18T16:07:49.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.302+0000 7f384c5a1700 1 Processor -- start 2023-10-18T16:07:49.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.302+0000 7f384c5a1700 1 -- start start 2023-10-18T16:07:49.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.303+0000 7f384c5a1700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f38440c20a0 0x7f3844154280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:49.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.303+0000 7f384c5a1700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3844150340 0x7f38441507f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:49.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.303+0000 7f384c5a1700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38440a8570 con 0x7f384414d880 2023-10-18T16:07:49.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.303+0000 7f384c5a1700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3844154770 con 0x7f3844150340 2023-10-18T16:07:49.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.303+0000 7f384c5a1700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38440a87c0 con 0x7f38440c20a0 2023-10-18T16:07:49.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.303+0000 7f384a33d700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f384414d880 0x7f38440c1990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:52474/0 (socket says 172.21.15.88:52474) 2023-10-18T16:07:49.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.303+0000 7f384a33d700 1 -- 172.21.15.88:0/3654664837 learned_addr learned my addr 172.21.15.88:0/3654664837 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:07:49.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.303+0000 7f384ab3e700 1 --2- 172.21.15.88:0/3654664837 >> v2:172.21.15.151:3300/0 conn(0x7f3844150340 0x7f38441507f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:49.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.303+0000 7f3849b3c700 1 --2- 172.21.15.88:0/3654664837 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f38440c20a0 0x7f3844154280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:49.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.303+0000 7f384933b700 1 -- 172.21.15.88:0/3654664837 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2917203141 0 0) 0x7f38440a8570 con 0x7f384414d880 2023-10-18T16:07:49.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.304+0000 7f384933b700 1 -- 172.21.15.88:0/3654664837 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38300033f0 con 0x7f384414d880 2023-10-18T16:07:49.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.304+0000 7f384933b700 1 -- 172.21.15.88:0/3654664837 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3540741979 0 0) 0x7f38300033f0 con 0x7f384414d880 2023-10-18T16:07:49.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.304+0000 7f384933b700 1 -- 172.21.15.88:0/3654664837 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f38440c20a0 msgr2=0x7f3844154280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:49.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.304+0000 7f384933b700 1 --2- 172.21.15.88:0/3654664837 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f38440c20a0 0x7f3844154280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:49.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.304+0000 7f384933b700 1 -- 172.21.15.88:0/3654664837 >> v2:172.21.15.151:3300/0 conn(0x7f3844150340 msgr2=0x7f38441507f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:49.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.304+0000 7f384933b700 1 --2- 172.21.15.88:0/3654664837 >> v2:172.21.15.151:3300/0 conn(0x7f3844150340 0x7f38441507f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:49.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.304+0000 7f384933b700 1 -- 172.21.15.88:0/3654664837 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38440a8aa0 con 0x7f384414d880 2023-10-18T16:07:49.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.304+0000 7f384933b700 1 -- 172.21.15.88:0/3654664837 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441715075 0 0) 0x7f3840001470 con 0x7f384414d880 2023-10-18T16:07:49.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.304+0000 7f384933b700 1 -- 172.21.15.88:0/3654664837 >> v1:172.21.15.88:6789/0 conn(0x7f384414d880 legacy=0x7f38440c1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:49.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.304+0000 7f384933b700 1 --2- 172.21.15.88:0/3654664837 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f38300038e0 0x7f3830003d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:49.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.304+0000 7f384933b700 1 --2- 172.21.15.88:0/3654664837 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f38440c20a0 0x7f3830004690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:49.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.304+0000 7f384933b700 1 --2- 172.21.15.88:0/3654664837 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3830004b80 0x7f3830005000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:49.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.304+0000 7f384933b700 1 -- 172.21.15.88:0/3654664837 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3844154770 con 0x7f38300038e0 2023-10-18T16:07:49.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.304+0000 7f384933b700 1 -- 172.21.15.88:0/3654664837 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38440a87c0 con 0x7f3830004b80 2023-10-18T16:07:49.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.304+0000 7f384933b700 1 -- 172.21.15.88:0/3654664837 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f383000b8e0 con 0x7f38440c20a0 2023-10-18T16:07:49.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.304+0000 7f384a33d700 1 --2- 172.21.15.88:0/3654664837 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f38300038e0 0x7f3830003d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:49.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.304+0000 7f384ab3e700 1 --2- 172.21.15.88:0/3654664837 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3830004b80 0x7f3830005000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:49.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.304+0000 7f3849b3c700 1 --2- 172.21.15.88:0/3654664837 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f38440c20a0 0x7f3830004690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:49.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.304+0000 7f384a33d700 1 -- 172.21.15.88:0/3654664837 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f38440c20a0 msgr2=0x7f3830004690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:49.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.304+0000 7f384a33d700 1 --2- 172.21.15.88:0/3654664837 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f38440c20a0 0x7f3830004690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:49.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.304+0000 7f384a33d700 1 -- 172.21.15.88:0/3654664837 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3830004b80 msgr2=0x7f3830005000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:49.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.304+0000 7f384a33d700 1 --2- 172.21.15.88:0/3654664837 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3830004b80 0x7f3830005000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:49.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.304+0000 7f384a33d700 1 -- 172.21.15.88:0/3654664837 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38300033f0 con 0x7f38300038e0 2023-10-18T16:07:49.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.304+0000 7f3849b3c700 1 --2- 172.21.15.88:0/3654664837 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f38440c20a0 0x7f3830004690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:07:49.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.304+0000 7f384a33d700 1 --2- 172.21.15.88:0/3654664837 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f38300038e0 0x7f3830003d00 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f384000c210 tx=0x7f3840001e90).ready entity=mon.0 client_cookie=9a4b7eabbba70b88 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:49.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.304+0000 7f384933b700 1 -- 172.21.15.88:0/3654664837 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3840007a60 con 0x7f38300038e0 2023-10-18T16:07:49.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.304+0000 7f384933b700 1 -- 172.21.15.88:0/3654664837 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f384000cae0 con 0x7f38300038e0 2023-10-18T16:07:49.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.304+0000 7f384933b700 1 -- 172.21.15.88:0/3654664837 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f384000ce60 con 0x7f38300038e0 2023-10-18T16:07:49.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.305+0000 7f384c5a1700 1 -- 172.21.15.88:0/3654664837 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f38300038e0 msgr2=0x7f3830003d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:49.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.305+0000 7f384c5a1700 1 --2- 172.21.15.88:0/3654664837 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f38300038e0 0x7f3830003d00 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f384000c210 tx=0x7f3840001e90).stop 2023-10-18T16:07:49.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.305+0000 7f384ab3e700 1 -- 172.21.15.88:0/3654664837 reap_dead start 2023-10-18T16:07:49.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.305+0000 7f384c5a1700 1 -- 172.21.15.88:0/3654664837 shutdown_connections 2023-10-18T16:07:49.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.305+0000 7f384c5a1700 1 -- 172.21.15.88:0/3654664837 >> 172.21.15.88:0/3654664837 conn(0x7f3844069d80 msgr2=0x7f384406a190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:49.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.305+0000 7f384c5a1700 1 -- 172.21.15.88:0/3654664837 shutdown_connections 2023-10-18T16:07:49.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.305+0000 7f384c5a1700 1 -- 172.21.15.88:0/3654664837 wait complete. 2023-10-18T16:07:49.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.306+0000 7f384c5a1700 1 Processor -- start 2023-10-18T16:07:49.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.306+0000 7f384c5a1700 1 -- start start 2023-10-18T16:07:49.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.307+0000 7f384c5a1700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f38300038e0 0x7f384415fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:49.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.307+0000 7f384c5a1700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f384414bee0 0x7f38441626d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:49.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.307+0000 7f384c5a1700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f384414c360 0x7f3844162bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:49.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.307+0000 7f384c5a1700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3844162240 con 0x7f384414c360 2023-10-18T16:07:49.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.307+0000 7f384c5a1700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38441630b0 con 0x7f38300038e0 2023-10-18T16:07:49.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.307+0000 7f384c5a1700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3844161350 con 0x7f384414bee0 2023-10-18T16:07:49.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.307+0000 7f384ab3e700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f384414c360 0x7f3844162bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:49.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.307+0000 7f384ab3e700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f384414c360 0x7f3844162bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59556/0 (socket says 172.21.15.88:59556) 2023-10-18T16:07:49.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.307+0000 7f384ab3e700 1 -- 172.21.15.88:0/831089787 learned_addr learned my addr 172.21.15.88:0/831089787 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:49.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.307+0000 7f3849b3c700 1 --2- 172.21.15.88:0/831089787 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f384414bee0 0x7f38441626d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:49.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.307+0000 7f384a33d700 1 --2- 172.21.15.88:0/831089787 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f38300038e0 0x7f384415fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:49.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.307+0000 7f384ab3e700 1 -- 172.21.15.88:0/831089787 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f384414bee0 msgr2=0x7f38441626d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:49.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.307+0000 7f384ab3e700 1 --2- 172.21.15.88:0/831089787 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f384414bee0 0x7f38441626d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:49.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.307+0000 7f384ab3e700 1 -- 172.21.15.88:0/831089787 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f38300038e0 msgr2=0x7f384415fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:49.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.307+0000 7f384ab3e700 1 --2- 172.21.15.88:0/831089787 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f38300038e0 0x7f384415fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:49.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.307+0000 7f384ab3e700 1 -- 172.21.15.88:0/831089787 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3844161630 con 0x7f384414c360 2023-10-18T16:07:49.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.307+0000 7f384ab3e700 1 --2- 172.21.15.88:0/831089787 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f384414c360 0x7f3844162bc0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f3844083b00 tx=0x7f383c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:49.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.307+0000 7f383a7fc700 1 -- 172.21.15.88:0/831089787 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f383c00e040 con 0x7f384414c360 2023-10-18T16:07:49.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.307+0000 7f383a7fc700 1 -- 172.21.15.88:0/831089787 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f383c00f040 con 0x7f384414c360 2023-10-18T16:07:49.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.307+0000 7f384c5a1700 1 -- 172.21.15.88:0/831089787 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3844161910 con 0x7f384414c360 2023-10-18T16:07:49.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.307+0000 7f383a7fc700 1 -- 172.21.15.88:0/831089787 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f383c00fd70 con 0x7f384414c360 2023-10-18T16:07:49.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.308+0000 7f384c5a1700 1 -- 172.21.15.88:0/831089787 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3844161ed0 con 0x7f384414c360 2023-10-18T16:07:49.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.308+0000 7f384c5a1700 1 -- 172.21.15.88:0/831089787 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f384406c4e0 con 0x7f384414c360 2023-10-18T16:07:49.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.309+0000 7f383a7fc700 1 -- 172.21.15.88:0/831089787 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f383c015660 con 0x7f384414c360 2023-10-18T16:07:49.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.309+0000 7f383a7fc700 1 --2- 172.21.15.88:0/831089787 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f382805f120 0x7f38280615e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:49.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.309+0000 7f383a7fc700 1 -- 172.21.15.88:0/831089787 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6121+0+0 (secure 0 0 0) 0x7f383c07aac0 con 0x7f384414c360 2023-10-18T16:07:49.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.309+0000 7f383a7fc700 1 --2- 172.21.15.88:0/831089787 >> [v2:172.21.15.88:6816/91008,v1:172.21.15.88:6817/91008] conn(0x7f38280684c0 0x7f382806a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:49.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.309+0000 7f383a7fc700 1 -- 172.21.15.88:0/831089787 --> [v2:172.21.15.88:6816/91008,v1:172.21.15.88:6817/91008] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f382806afa0 con 0x7f38280684c0 2023-10-18T16:07:49.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.309+0000 7f383a7fc700 1 -- 172.21.15.88:0/831089787 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f383c07f220 con 0x7f384414c360 2023-10-18T16:07:49.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.309+0000 7f384a33d700 1 --2- 172.21.15.88:0/831089787 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f382805f120 0x7f38280615e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:49.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.309+0000 7f3849b3c700 1 --2- 172.21.15.88:0/831089787 >> [v2:172.21.15.88:6816/91008,v1:172.21.15.88:6817/91008] conn(0x7f38280684c0 0x7f382806a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:49.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.310+0000 7f3849b3c700 1 --2- 172.21.15.88:0/831089787 >> [v2:172.21.15.88:6816/91008,v1:172.21.15.88:6817/91008] conn(0x7f38280684c0 0x7f382806a920 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:49.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.310+0000 7f384a33d700 1 --2- 172.21.15.88:0/831089787 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f382805f120 0x7f38280615e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f3840001ed0 tx=0x7f3840015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:49.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.310+0000 7f383a7fc700 1 -- 172.21.15.88:0/831089787 <== osd.3 v2:172.21.15.88:6816/91008 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23065 (crc 0 0 0) 0x7f382806afa0 con 0x7f38280684c0 2023-10-18T16:07:49.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.324+0000 7f384c5a1700 1 -- 172.21.15.88:0/831089787 --> [v2:172.21.15.88:6816/91008,v1:172.21.15.88:6817/91008] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f38440bd7a0 con 0x7f38280684c0 2023-10-18T16:07:49.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.325+0000 7f383a7fc700 1 -- 172.21.15.88:0/831089787 <== osd.3 v2:172.21.15.88:6816/91008 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f38440bd7a0 con 0x7f38280684c0 2023-10-18T16:07:49.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.325+0000 7f384c5a1700 1 -- 172.21.15.88:0/831089787 >> [v2:172.21.15.88:6816/91008,v1:172.21.15.88:6817/91008] conn(0x7f38280684c0 msgr2=0x7f382806a920 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:49.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.325+0000 7f384c5a1700 1 --2- 172.21.15.88:0/831089787 >> [v2:172.21.15.88:6816/91008,v1:172.21.15.88:6817/91008] conn(0x7f38280684c0 0x7f382806a920 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:49.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.325+0000 7f384c5a1700 1 -- 172.21.15.88:0/831089787 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f382805f120 msgr2=0x7f38280615e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:49.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.325+0000 7f384c5a1700 1 --2- 172.21.15.88:0/831089787 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f382805f120 0x7f38280615e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f3840001ed0 tx=0x7f3840015000).stop 2023-10-18T16:07:49.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.325+0000 7f384c5a1700 1 -- 172.21.15.88:0/831089787 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f384414c360 msgr2=0x7f3844162bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:49.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.325+0000 7f384c5a1700 1 --2- 172.21.15.88:0/831089787 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f384414c360 0x7f3844162bc0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f3844083b00 tx=0x7f383c00b040).stop 2023-10-18T16:07:49.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.325+0000 7f384ab3e700 1 -- 172.21.15.88:0/831089787 reap_dead start 2023-10-18T16:07:49.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.325+0000 7f384c5a1700 1 -- 172.21.15.88:0/831089787 shutdown_connections 2023-10-18T16:07:49.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.325+0000 7f384c5a1700 1 -- 172.21.15.88:0/831089787 >> 172.21.15.88:0/831089787 conn(0x7f3844069d80 msgr2=0x7f384414f610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:49.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.325+0000 7f384c5a1700 1 -- 172.21.15.88:0/831089787 shutdown_connections 2023-10-18T16:07:49.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.325+0000 7f384c5a1700 1 -- 172.21.15.88:0/831089787 wait complete. 2023-10-18T16:07:49.340 INFO:teuthology.orchestra.run.smithi088.stdout:55834574854 2023-10-18T16:07:49.340 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-10-18T16:07:49.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.454+0000 7ff354522700 1 Processor -- start 2023-10-18T16:07:49.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.454+0000 7ff354522700 1 -- start start 2023-10-18T16:07:49.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.455+0000 7ff354522700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7ff34c0bcc30 0x7ff34c0bd010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:49.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.455+0000 7ff354522700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff34c0bd8d0 0x7ff34c064870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:49.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.455+0000 7ff354522700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff34c0b9d80 con 0x7ff34c062730 2023-10-18T16:07:49.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.455+0000 7ff354522700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff34c0be770 con 0x7ff34c0bcc30 2023-10-18T16:07:49.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.455+0000 7ff354522700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff34c150b80 con 0x7ff34c0bd8d0 2023-10-18T16:07:49.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.455+0000 7ff3522be700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7ff34c0bcc30 0x7ff34c0bd010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:49.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.455+0000 7ff352abf700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff34c0bd8d0 0x7ff34c064870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:49.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.455+0000 7ff351abd700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7ff34c062730 0x7ff34c062e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.88:52586/0 (socket says 172.21.15.88:52586) 2023-10-18T16:07:49.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.455+0000 7ff351abd700 1 -- 172.21.15.88:0/3198924697 learned_addr learned my addr 172.21.15.88:0/3198924697 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:07:49.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.455+0000 7ff352abf700 1 -- 172.21.15.88:0/3198924697 >> v2:172.21.15.151:3300/0 conn(0x7ff34c0bcc30 msgr2=0x7ff34c0bd010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:49.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.455+0000 7ff352abf700 1 --2- 172.21.15.88:0/3198924697 >> v2:172.21.15.151:3300/0 conn(0x7ff34c0bcc30 0x7ff34c0bd010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:49.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.455+0000 7ff352abf700 1 -- 172.21.15.88:0/3198924697 >> v1:172.21.15.151:6789/0 conn(0x7ff34c062730 legacy=0x7ff34c062e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:49.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.455+0000 7ff352abf700 1 -- 172.21.15.88:0/3198924697 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff34c1502b0 con 0x7ff34c0bd8d0 2023-10-18T16:07:49.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.455+0000 7ff3522be700 1 --2- 172.21.15.88:0/3198924697 >> v2:172.21.15.151:3300/0 conn(0x7ff34c0bcc30 0x7ff34c0bd010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2023-10-18T16:07:49.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.455+0000 7ff352abf700 1 --2- 172.21.15.88:0/3198924697 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff34c0bd8d0 0x7ff34c064870 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7ff344009e60 tx=0x7ff344009780).ready entity=mon.2 client_cookie=a41f3c1deccd2e90 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:49.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.460+0000 7ff3512bc700 1 -- 172.21.15.88:0/3198924697 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff34400e070 con 0x7ff34c0bd8d0 2023-10-18T16:07:49.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.460+0000 7ff3512bc700 1 -- 172.21.15.88:0/3198924697 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff34400bb60 con 0x7ff34c0bd8d0 2023-10-18T16:07:49.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.460+0000 7ff3512bc700 1 -- 172.21.15.88:0/3198924697 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff344012410 con 0x7ff34c0bd8d0 2023-10-18T16:07:49.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.461+0000 7ff354522700 1 -- 172.21.15.88:0/3198924697 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff34c0bd8d0 msgr2=0x7ff34c064870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:49.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.461+0000 7ff354522700 1 --2- 172.21.15.88:0/3198924697 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff34c0bd8d0 0x7ff34c064870 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7ff344009e60 tx=0x7ff344009780).stop 2023-10-18T16:07:49.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.461+0000 7ff354522700 1 -- 172.21.15.88:0/3198924697 shutdown_connections 2023-10-18T16:07:49.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.461+0000 7ff354522700 1 --2- 172.21.15.88:0/3198924697 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff34c0bd8d0 0x7ff34c064870 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:49.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.461+0000 7ff354522700 1 --2- 172.21.15.88:0/3198924697 >> v2:172.21.15.151:3300/0 conn(0x7ff34c0bcc30 0x7ff34c0bd010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:49.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.461+0000 7ff354522700 1 -- 172.21.15.88:0/3198924697 >> 172.21.15.88:0/3198924697 conn(0x7ff34c069360 msgr2=0x7ff34c069770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:49.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.461+0000 7ff354522700 1 -- 172.21.15.88:0/3198924697 shutdown_connections 2023-10-18T16:07:49.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.461+0000 7ff354522700 1 -- 172.21.15.88:0/3198924697 wait complete. 2023-10-18T16:07:49.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.462+0000 7ff354522700 1 Processor -- start 2023-10-18T16:07:49.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.462+0000 7ff354522700 1 -- start start 2023-10-18T16:07:49.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.463+0000 7ff354522700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff34c062730 0x7ff34c147f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:49.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.463+0000 7ff354522700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff34c0bcc30 0x7ff34c163e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:49.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.463+0000 7ff354522700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff34c0bd8d0 0x7ff34c14a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:49.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.463+0000 7ff354522700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff34c14ba30 con 0x7ff34c0bd8d0 2023-10-18T16:07:49.471 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.463+0000 7ff354522700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff34c163700 con 0x7ff34c0bcc30 2023-10-18T16:07:49.471 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.463+0000 7ff354522700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff34c0b9590 con 0x7ff34c062730 2023-10-18T16:07:49.471 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.463+0000 7ff352abf700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff34c0bd8d0 0x7ff34c14a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:49.471 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.463+0000 7ff352abf700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff34c0bd8d0 0x7ff34c14a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59568/0 (socket says 172.21.15.88:59568) 2023-10-18T16:07:49.471 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.463+0000 7ff352abf700 1 -- 172.21.15.88:0/3813629309 learned_addr learned my addr 172.21.15.88:0/3813629309 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:49.471 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.463+0000 7ff3522be700 1 --2- 172.21.15.88:0/3813629309 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff34c062730 0x7ff34c147f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:49.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.463+0000 7ff351abd700 1 --2- 172.21.15.88:0/3813629309 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff34c0bcc30 0x7ff34c163e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:49.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.463+0000 7ff352abf700 1 -- 172.21.15.88:0/3813629309 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff34c062730 msgr2=0x7ff34c147f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:49.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.463+0000 7ff352abf700 1 --2- 172.21.15.88:0/3813629309 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff34c062730 0x7ff34c147f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:49.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.463+0000 7ff352abf700 1 -- 172.21.15.88:0/3813629309 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff34c0bcc30 msgr2=0x7ff34c163e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:49.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.463+0000 7ff352abf700 1 --2- 172.21.15.88:0/3813629309 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff34c0bcc30 0x7ff34c163e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:49.473 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.463+0000 7ff352abf700 1 -- 172.21.15.88:0/3813629309 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3480092a0 con 0x7ff34c0bd8d0 2023-10-18T16:07:49.473 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.463+0000 7ff352abf700 1 --2- 172.21.15.88:0/3813629309 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff34c0bd8d0 0x7ff34c14a9c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7ff34c0be8b0 tx=0x7ff34c0ba550).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:49.473 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.464+0000 7ff3427fc700 1 -- 172.21.15.88:0/3813629309 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff344012410 con 0x7ff34c0bd8d0 2023-10-18T16:07:49.473 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.464+0000 7ff3427fc700 1 -- 172.21.15.88:0/3813629309 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff34401d630 con 0x7ff34c0bd8d0 2023-10-18T16:07:49.473 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.464+0000 7ff354522700 1 -- 172.21.15.88:0/3813629309 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff344009a20 con 0x7ff34c0bd8d0 2023-10-18T16:07:49.473 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.464+0000 7ff3427fc700 1 -- 172.21.15.88:0/3813629309 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff34401d9f0 con 0x7ff34c0bd8d0 2023-10-18T16:07:49.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.464+0000 7ff354522700 1 -- 172.21.15.88:0/3813629309 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff34c161640 con 0x7ff34c0bd8d0 2023-10-18T16:07:49.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.464+0000 7ff354522700 1 -- 172.21.15.88:0/3813629309 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff34c06c4e0 con 0x7ff34c0bd8d0 2023-10-18T16:07:49.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.465+0000 7ff3427fc700 1 -- 172.21.15.88:0/3813629309 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7ff344003040 con 0x7ff34c0bd8d0 2023-10-18T16:07:49.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.465+0000 7ff3427fc700 1 --2- 172.21.15.88:0/3813629309 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7ff33005f180 0x7ff330061640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:49.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.465+0000 7ff3427fc700 1 -- 172.21.15.88:0/3813629309 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6121+0+0 (secure 0 0 0) 0x7ff34407c940 con 0x7ff34c0bd8d0 2023-10-18T16:07:49.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.465+0000 7ff3427fc700 1 --2- 172.21.15.88:0/3813629309 >> [v2:172.21.15.151:6800/91514,v1:172.21.15.151:6801/91514] conn(0x7ff330068520 0x7ff33006a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:49.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.465+0000 7ff3427fc700 1 -- 172.21.15.88:0/3813629309 --> [v2:172.21.15.151:6800/91514,v1:172.21.15.151:6801/91514] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff33006b000 con 0x7ff330068520 2023-10-18T16:07:49.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.465+0000 7ff3522be700 1 --2- 172.21.15.88:0/3813629309 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7ff33005f180 0x7ff330061640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:49.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.465+0000 7ff3427fc700 1 -- 172.21.15.88:0/3813629309 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff34407ccf0 con 0x7ff34c0bd8d0 2023-10-18T16:07:49.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.466+0000 7ff351abd700 1 --2- 172.21.15.88:0/3813629309 >> [v2:172.21.15.151:6800/91514,v1:172.21.15.151:6801/91514] conn(0x7ff330068520 0x7ff33006a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:49.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.466+0000 7ff3522be700 1 --2- 172.21.15.88:0/3813629309 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7ff33005f180 0x7ff330061640 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7ff348009dc0 tx=0x7ff348009060).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:49.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.466+0000 7ff351abd700 1 --2- 172.21.15.88:0/3813629309 >> [v2:172.21.15.151:6800/91514,v1:172.21.15.151:6801/91514] conn(0x7ff330068520 0x7ff33006a980 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:49.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.467+0000 7ff3427fc700 1 -- 172.21.15.88:0/3813629309 <== osd.4 v2:172.21.15.151:6800/91514 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23065 (crc 0 0 0) 0x7ff33006b000 con 0x7ff330068520 2023-10-18T16:07:49.484 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.481+0000 7ff354522700 1 -- 172.21.15.88:0/3813629309 --> [v2:172.21.15.151:6800/91514,v1:172.21.15.151:6801/91514] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff34c067b20 con 0x7ff330068520 2023-10-18T16:07:49.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.481+0000 7ff3427fc700 1 -- 172.21.15.88:0/3813629309 <== osd.4 v2:172.21.15.151:6800/91514 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ff34c067b20 con 0x7ff330068520 2023-10-18T16:07:49.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.481+0000 7ff354522700 1 -- 172.21.15.88:0/3813629309 >> [v2:172.21.15.151:6800/91514,v1:172.21.15.151:6801/91514] conn(0x7ff330068520 msgr2=0x7ff33006a980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:49.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.481+0000 7ff354522700 1 --2- 172.21.15.88:0/3813629309 >> [v2:172.21.15.151:6800/91514,v1:172.21.15.151:6801/91514] conn(0x7ff330068520 0x7ff33006a980 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:49.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.481+0000 7ff354522700 1 -- 172.21.15.88:0/3813629309 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7ff33005f180 msgr2=0x7ff330061640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:49.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.481+0000 7ff354522700 1 --2- 172.21.15.88:0/3813629309 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7ff33005f180 0x7ff330061640 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7ff348009dc0 tx=0x7ff348009060).stop 2023-10-18T16:07:49.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.481+0000 7ff354522700 1 -- 172.21.15.88:0/3813629309 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff34c0bd8d0 msgr2=0x7ff34c14a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:49.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.481+0000 7ff354522700 1 --2- 172.21.15.88:0/3813629309 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff34c0bd8d0 0x7ff34c14a9c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7ff34c0be8b0 tx=0x7ff34c0ba550).stop 2023-10-18T16:07:49.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.481+0000 7ff352abf700 1 -- 172.21.15.88:0/3813629309 reap_dead start 2023-10-18T16:07:49.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.481+0000 7ff354522700 1 -- 172.21.15.88:0/3813629309 shutdown_connections 2023-10-18T16:07:49.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.482+0000 7ff354522700 1 -- 172.21.15.88:0/3813629309 >> 172.21.15.88:0/3813629309 conn(0x7ff34c069360 msgr2=0x7ff34c063640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:49.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.482+0000 7ff354522700 1 -- 172.21.15.88:0/3813629309 shutdown_connections 2023-10-18T16:07:49.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.482+0000 7ff354522700 1 -- 172.21.15.88:0/3813629309 wait complete. 2023-10-18T16:07:49.496 INFO:teuthology.orchestra.run.smithi088.stdout:55834574854 2023-10-18T16:07:49.497 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-10-18T16:07:49.580 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2023-10-18T16:07:49.610 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.606+0000 7f0bc1f5e700 1 Processor -- start 2023-10-18T16:07:49.610 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.606+0000 7f0bc1f5e700 1 -- start start 2023-10-18T16:07:49.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.607+0000 7f0bc1f5e700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f0bbc14d880 0x7f0bbc154280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:49.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.607+0000 7f0bc1f5e700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0bbc0c1990 0x7f0bbc0c1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:49.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.607+0000 7f0bc1f5e700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bbc0a8570 con 0x7f0bbc064490 2023-10-18T16:07:49.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.607+0000 7f0bc1f5e700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bbc154770 con 0x7f0bbc0c1990 2023-10-18T16:07:49.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.607+0000 7f0bc1f5e700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bbc0a87c0 con 0x7f0bbc14d880 2023-10-18T16:07:49.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.607+0000 7f0bbaffd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0bbc0c1990 0x7f0bbc0c1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:49.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.607+0000 7f0bbbfff700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f0bbc064490 0x7f0bbc0648c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:52486/0 (socket says 172.21.15.88:52486) 2023-10-18T16:07:49.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.607+0000 7f0bbbfff700 1 -- 172.21.15.88:0/3100180620 learned_addr learned my addr 172.21.15.88:0/3100180620 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:07:49.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.607+0000 7f0bbaffd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0bbc0c1990 0x7f0bbc0c1df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59570/0 (socket says 172.21.15.88:59570) 2023-10-18T16:07:49.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.608+0000 7f0bbb7fe700 1 --2- 172.21.15.88:0/3100180620 >> v2:172.21.15.151:3300/0 conn(0x7f0bbc14d880 0x7f0bbc154280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:49.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.608+0000 7f0bbaffd700 1 -- 172.21.15.88:0/3100180620 >> v2:172.21.15.151:3300/0 conn(0x7f0bbc14d880 msgr2=0x7f0bbc154280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:49.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.608+0000 7f0bbaffd700 1 --2- 172.21.15.88:0/3100180620 >> v2:172.21.15.151:3300/0 conn(0x7f0bbc14d880 0x7f0bbc154280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:49.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.608+0000 7f0bbaffd700 1 -- 172.21.15.88:0/3100180620 >> v1:172.21.15.88:6789/0 conn(0x7f0bbc064490 legacy=0x7f0bbc0648c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:49.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.608+0000 7f0bba7fc700 1 -- 172.21.15.88:0/3100180620 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3845122595 0 0) 0x7f0bbc0a8570 con 0x7f0bbc064490 2023-10-18T16:07:49.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.608+0000 7f0bbaffd700 1 -- 172.21.15.88:0/3100180620 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bbc0a8aa0 con 0x7f0bbc0c1990 2023-10-18T16:07:49.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.608+0000 7f0bbb7fe700 1 --2- 172.21.15.88:0/3100180620 >> v2:172.21.15.151:3300/0 conn(0x7f0bbc14d880 0x7f0bbc154280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-10-18T16:07:49.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.608+0000 7f0bbaffd700 1 --2- 172.21.15.88:0/3100180620 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0bbc0c1990 0x7f0bbc0c1df0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f0bbc062ae0 tx=0x7f0bb0009060).ready entity=mon.0 client_cookie=a3d45d65072fe03 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:49.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.608+0000 7f0bba7fc700 1 -- 172.21.15.88:0/3100180620 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb000b580 con 0x7f0bbc0c1990 2023-10-18T16:07:49.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.608+0000 7f0bba7fc700 1 -- 172.21.15.88:0/3100180620 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bb000b6e0 con 0x7f0bbc0c1990 2023-10-18T16:07:49.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.608+0000 7f0bba7fc700 1 -- 172.21.15.88:0/3100180620 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb000ba60 con 0x7f0bbc0c1990 2023-10-18T16:07:49.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.609+0000 7f0bc1f5e700 1 -- 172.21.15.88:0/3100180620 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0bbc0c1990 msgr2=0x7f0bbc0c1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:49.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.609+0000 7f0bc1f5e700 1 --2- 172.21.15.88:0/3100180620 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0bbc0c1990 0x7f0bbc0c1df0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f0bbc062ae0 tx=0x7f0bb0009060).stop 2023-10-18T16:07:49.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.609+0000 7f0bc1f5e700 1 -- 172.21.15.88:0/3100180620 shutdown_connections 2023-10-18T16:07:49.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.609+0000 7f0bc1f5e700 1 --2- 172.21.15.88:0/3100180620 >> v2:172.21.15.151:3300/0 conn(0x7f0bbc14d880 0x7f0bbc154280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:49.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.609+0000 7f0bc1f5e700 1 --2- 172.21.15.88:0/3100180620 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0bbc0c1990 0x7f0bbc0c1df0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:49.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.609+0000 7f0bc1f5e700 1 -- 172.21.15.88:0/3100180620 >> 172.21.15.88:0/3100180620 conn(0x7f0bbc069d80 msgr2=0x7f0bbc06a190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:49.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.609+0000 7f0bc1f5e700 1 -- 172.21.15.88:0/3100180620 shutdown_connections 2023-10-18T16:07:49.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.609+0000 7f0bc1f5e700 1 -- 172.21.15.88:0/3100180620 wait complete. 2023-10-18T16:07:49.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.610+0000 7f0bc1f5e700 1 Processor -- start 2023-10-18T16:07:49.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.610+0000 7f0bc1f5e700 1 -- start start 2023-10-18T16:07:49.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.611+0000 7f0bc1f5e700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0bbc064490 0x7f0bbc148070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:49.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.611+0000 7f0bc1f5e700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bbc0c1990 0x7f0bbc152780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:49.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.611+0000 7f0bc1f5e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0bbc14d880 0x7f0bbc152c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:49.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.611+0000 7f0bc1f5e700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bbc0bcd60 con 0x7f0bbc064490 2023-10-18T16:07:49.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.611+0000 7f0bc1f5e700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bbc0ab960 con 0x7f0bbc14d880 2023-10-18T16:07:49.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.611+0000 7f0bc1f5e700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bbc1510a0 con 0x7f0bbc0c1990 2023-10-18T16:07:49.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.611+0000 7f0bbb7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0bbc064490 0x7f0bbc148070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:49.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.611+0000 7f0bbb7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0bbc064490 0x7f0bbc148070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59574/0 (socket says 172.21.15.88:59574) 2023-10-18T16:07:49.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.611+0000 7f0bbb7fe700 1 -- 172.21.15.88:0/1431889084 learned_addr learned my addr 172.21.15.88:0/1431889084 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:49.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.611+0000 7f0bbaffd700 1 --2- 172.21.15.88:0/1431889084 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bbc0c1990 0x7f0bbc152780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:49.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.611+0000 7f0bbbfff700 1 --2- 172.21.15.88:0/1431889084 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0bbc14d880 0x7f0bbc152c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:49.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.611+0000 7f0bbb7fe700 1 -- 172.21.15.88:0/1431889084 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bbc0c1990 msgr2=0x7f0bbc152780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:49.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.611+0000 7f0bbb7fe700 1 --2- 172.21.15.88:0/1431889084 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bbc0c1990 0x7f0bbc152780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:49.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.611+0000 7f0bbb7fe700 1 -- 172.21.15.88:0/1431889084 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0bbc14d880 msgr2=0x7f0bbc152c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:49.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.611+0000 7f0bbb7fe700 1 --2- 172.21.15.88:0/1431889084 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0bbc14d880 0x7f0bbc152c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:49.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.611+0000 7f0bbb7fe700 1 -- 172.21.15.88:0/1431889084 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bb00092a0 con 0x7f0bbc064490 2023-10-18T16:07:49.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.611+0000 7f0bbbfff700 1 --2- 172.21.15.88:0/1431889084 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0bbc14d880 0x7f0bbc152c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:07:49.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.611+0000 7f0bbb7fe700 1 --2- 172.21.15.88:0/1431889084 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0bbc064490 0x7f0bbc148070 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f0bac007f90 tx=0x7f0bac007870).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:49.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.611+0000 7f0ba37fe700 1 -- 172.21.15.88:0/1431889084 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bac016400 con 0x7f0bbc064490 2023-10-18T16:07:49.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.611+0000 7f0ba37fe700 1 -- 172.21.15.88:0/1431889084 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bac016560 con 0x7f0bbc064490 2023-10-18T16:07:49.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.611+0000 7f0bc1f5e700 1 -- 172.21.15.88:0/1431889084 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bbc0abbb0 con 0x7f0bbc064490 2023-10-18T16:07:49.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.611+0000 7f0ba37fe700 1 -- 172.21.15.88:0/1431889084 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bac0168e0 con 0x7f0bbc064490 2023-10-18T16:07:49.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.611+0000 7f0bc1f5e700 1 -- 172.21.15.88:0/1431889084 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bbc0ac170 con 0x7f0bbc064490 2023-10-18T16:07:49.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.612+0000 7f0bc1f5e700 1 -- 172.21.15.88:0/1431889084 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0bbc06c4e0 con 0x7f0bbc064490 2023-10-18T16:07:49.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.612+0000 7f0ba37fe700 1 -- 172.21.15.88:0/1431889084 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f0bac01f3f0 con 0x7f0bbc064490 2023-10-18T16:07:49.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.613+0000 7f0ba37fe700 1 --2- 172.21.15.88:0/1431889084 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f0ba805f150 0x7f0ba8061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:49.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.613+0000 7f0ba37fe700 1 -- 172.21.15.88:0/1431889084 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6121+0+0 (secure 0 0 0) 0x7f0bac07acd0 con 0x7f0bbc064490 2023-10-18T16:07:49.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.613+0000 7f0ba37fe700 1 --2- 172.21.15.88:0/1431889084 >> [v2:172.21.15.151:6808/91513,v1:172.21.15.151:6809/91513] conn(0x7f0ba80684f0 0x7f0ba806a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:49.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.613+0000 7f0bbaffd700 1 --2- 172.21.15.88:0/1431889084 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f0ba805f150 0x7f0ba8061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:49.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.613+0000 7f0ba37fe700 1 -- 172.21.15.88:0/1431889084 --> [v2:172.21.15.151:6808/91513,v1:172.21.15.151:6809/91513] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0ba806afd0 con 0x7f0ba80684f0 2023-10-18T16:07:49.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.613+0000 7f0ba37fe700 1 -- 172.21.15.88:0/1431889084 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0bac07d2f0 con 0x7f0bbc064490 2023-10-18T16:07:49.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.613+0000 7f0bbbfff700 1 --2- 172.21.15.88:0/1431889084 >> [v2:172.21.15.151:6808/91513,v1:172.21.15.151:6809/91513] conn(0x7f0ba80684f0 0x7f0ba806a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:49.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.613+0000 7f0bbaffd700 1 --2- 172.21.15.88:0/1431889084 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f0ba805f150 0x7f0ba8061610 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f0bbc0c1080 tx=0x7f0bb0017000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:49.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.613+0000 7f0bbbfff700 1 --2- 172.21.15.88:0/1431889084 >> [v2:172.21.15.151:6808/91513,v1:172.21.15.151:6809/91513] conn(0x7f0ba80684f0 0x7f0ba806a950 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:49.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.614+0000 7f0ba37fe700 1 -- 172.21.15.88:0/1431889084 <== osd.5 v2:172.21.15.151:6808/91513 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23065 (crc 0 0 0) 0x7f0ba806afd0 con 0x7f0ba80684f0 2023-10-18T16:07:49.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.628+0000 7f0bc1f5e700 1 -- 172.21.15.88:0/1431889084 --> [v2:172.21.15.151:6808/91513,v1:172.21.15.151:6809/91513] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0bbc0a8d80 con 0x7f0ba80684f0 2023-10-18T16:07:49.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.629+0000 7f0ba37fe700 1 -- 172.21.15.88:0/1431889084 <== osd.5 v2:172.21.15.151:6808/91513 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f0bbc0a8d80 con 0x7f0ba80684f0 2023-10-18T16:07:49.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.629+0000 7f0bc1f5e700 1 -- 172.21.15.88:0/1431889084 >> [v2:172.21.15.151:6808/91513,v1:172.21.15.151:6809/91513] conn(0x7f0ba80684f0 msgr2=0x7f0ba806a950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:49.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.629+0000 7f0bc1f5e700 1 --2- 172.21.15.88:0/1431889084 >> [v2:172.21.15.151:6808/91513,v1:172.21.15.151:6809/91513] conn(0x7f0ba80684f0 0x7f0ba806a950 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:49.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.629+0000 7f0bc1f5e700 1 -- 172.21.15.88:0/1431889084 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f0ba805f150 msgr2=0x7f0ba8061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:49.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.629+0000 7f0bc1f5e700 1 --2- 172.21.15.88:0/1431889084 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f0ba805f150 0x7f0ba8061610 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f0bbc0c1080 tx=0x7f0bb0017000).stop 2023-10-18T16:07:49.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.629+0000 7f0bc1f5e700 1 -- 172.21.15.88:0/1431889084 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0bbc064490 msgr2=0x7f0bbc148070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:49.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.629+0000 7f0bc1f5e700 1 --2- 172.21.15.88:0/1431889084 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0bbc064490 0x7f0bbc148070 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f0bac007f90 tx=0x7f0bac007870).stop 2023-10-18T16:07:49.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.629+0000 7f0bbbfff700 1 -- 172.21.15.88:0/1431889084 reap_dead start 2023-10-18T16:07:49.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.629+0000 7f0bc1f5e700 1 -- 172.21.15.88:0/1431889084 shutdown_connections 2023-10-18T16:07:49.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.629+0000 7f0bc1f5e700 1 -- 172.21.15.88:0/1431889084 >> 172.21.15.88:0/1431889084 conn(0x7f0bbc069d80 msgr2=0x7f0bbc148a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:49.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.629+0000 7f0bc1f5e700 1 -- 172.21.15.88:0/1431889084 shutdown_connections 2023-10-18T16:07:49.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.629+0000 7f0bc1f5e700 1 -- 172.21.15.88:0/1431889084 wait complete. 2023-10-18T16:07:49.644 INFO:teuthology.orchestra.run.smithi088.stdout:55834574853 2023-10-18T16:07:49.644 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-10-18T16:07:49.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.755+0000 7fbaedc9d700 1 Processor -- start 2023-10-18T16:07:49.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.756+0000 7fbaedc9d700 1 -- start start 2023-10-18T16:07:49.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.756+0000 7fbaedc9d700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fbae80c38c0 0x7fbae80c3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:49.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.756+0000 7fbaedc9d700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbae80c0b70 0x7fbae8149c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:49.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.756+0000 7fbaedc9d700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbae814d280 con 0x7fbae806a670 2023-10-18T16:07:49.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.756+0000 7fbaedc9d700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbae8067520 con 0x7fbae80c0b70 2023-10-18T16:07:49.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.756+0000 7fbaedc9d700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbae8151530 con 0x7fbae80c38c0 2023-10-18T16:07:49.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.756+0000 7fbae7fff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbae80c0b70 0x7fbae8149c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:49.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.757+0000 7fbae7fff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbae80c0b70 0x7fbae8149c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59584/0 (socket says 172.21.15.88:59584) 2023-10-18T16:07:49.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.757+0000 7fbae7fff700 1 -- 172.21.15.88:0/3741456032 learned_addr learned my addr 172.21.15.88:0/3741456032 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:49.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.757+0000 7fbae6ffd700 1 --1- 172.21.15.88:0/3741456032 >> v1:172.21.15.88:6789/0 conn(0x7fbae806a670 0x7fbae814d710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:52498/0 (socket says 172.21.15.88:52498) 2023-10-18T16:07:49.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.757+0000 7fbae77fe700 1 --2- 172.21.15.88:0/3741456032 >> v2:172.21.15.151:3300/0 conn(0x7fbae80c38c0 0x7fbae80c3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:49.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.757+0000 7fbae7fff700 1 -- 172.21.15.88:0/3741456032 >> v2:172.21.15.151:3300/0 conn(0x7fbae80c38c0 msgr2=0x7fbae80c3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:49.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.757+0000 7fbae7fff700 1 --2- 172.21.15.88:0/3741456032 >> v2:172.21.15.151:3300/0 conn(0x7fbae80c38c0 0x7fbae80c3ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:49.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.757+0000 7fbae7fff700 1 -- 172.21.15.88:0/3741456032 >> v1:172.21.15.88:6789/0 conn(0x7fbae806a670 legacy=0x7fbae814d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:49.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.757+0000 7fbae67fc700 1 -- 172.21.15.88:0/3741456032 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3025938793 0 0) 0x7fbae814d280 con 0x7fbae806a670 2023-10-18T16:07:49.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.757+0000 7fbae7fff700 1 -- 172.21.15.88:0/3741456032 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbae81509d0 con 0x7fbae80c0b70 2023-10-18T16:07:49.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.757+0000 7fbae7fff700 1 --2- 172.21.15.88:0/3741456032 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbae80c0b70 0x7fbae8149c30 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fbae8066820 tx=0x7fbae000b3b0).ready entity=mon.0 client_cookie=fb51e9d21057e0a1 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:49.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.757+0000 7fbae67fc700 1 -- 172.21.15.88:0/3741456032 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae000db10 con 0x7fbae80c0b70 2023-10-18T16:07:49.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.757+0000 7fbae67fc700 1 -- 172.21.15.88:0/3741456032 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbae000dc70 con 0x7fbae80c0b70 2023-10-18T16:07:49.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.757+0000 7fbae67fc700 1 -- 172.21.15.88:0/3741456032 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae0010470 con 0x7fbae80c0b70 2023-10-18T16:07:49.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.757+0000 7fbaedc9d700 1 -- 172.21.15.88:0/3741456032 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbae80c0b70 msgr2=0x7fbae8149c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:49.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.757+0000 7fbaedc9d700 1 --2- 172.21.15.88:0/3741456032 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbae80c0b70 0x7fbae8149c30 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fbae8066820 tx=0x7fbae000b3b0).stop 2023-10-18T16:07:49.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.758+0000 7fbaedc9d700 1 -- 172.21.15.88:0/3741456032 shutdown_connections 2023-10-18T16:07:49.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.758+0000 7fbaedc9d700 1 --2- 172.21.15.88:0/3741456032 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbae80c0b70 0x7fbae8149c30 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:49.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.758+0000 7fbaedc9d700 1 --2- 172.21.15.88:0/3741456032 >> v2:172.21.15.151:3300/0 conn(0x7fbae80c38c0 0x7fbae80c3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:49.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.758+0000 7fbaedc9d700 1 -- 172.21.15.88:0/3741456032 >> 172.21.15.88:0/3741456032 conn(0x7fbae8065cd0 msgr2=0x7fbae80660e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:49.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.758+0000 7fbaedc9d700 1 -- 172.21.15.88:0/3741456032 shutdown_connections 2023-10-18T16:07:49.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.758+0000 7fbaedc9d700 1 -- 172.21.15.88:0/3741456032 wait complete. 2023-10-18T16:07:49.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.759+0000 7fbaedc9d700 1 Processor -- start 2023-10-18T16:07:49.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.759+0000 7fbaedc9d700 1 -- start start 2023-10-18T16:07:49.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.759+0000 7fbaedc9d700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbae806a670 0x7fbae814a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:49.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.759+0000 7fbaedc9d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbae80c0b70 0x7fbae81455b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:49.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.759+0000 7fbaedc9d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbae80c38c0 0x7fbae8153090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:49.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.760+0000 7fbaedc9d700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae8149870 con 0x7fbae80c0b70 2023-10-18T16:07:49.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.760+0000 7fbaedc9d700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae815efc0 con 0x7fbae80c38c0 2023-10-18T16:07:49.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.760+0000 7fbaedc9d700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbae815f860 con 0x7fbae806a670 2023-10-18T16:07:49.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.760+0000 7fbae6ffd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbae80c0b70 0x7fbae81455b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:49.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.760+0000 7fbae6ffd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbae80c0b70 0x7fbae81455b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59600/0 (socket says 172.21.15.88:59600) 2023-10-18T16:07:49.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.760+0000 7fbae6ffd700 1 -- 172.21.15.88:0/2729654831 learned_addr learned my addr 172.21.15.88:0/2729654831 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:49.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.760+0000 7fbae77fe700 1 --2- 172.21.15.88:0/2729654831 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbae806a670 0x7fbae814a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:49.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.760+0000 7fbae7fff700 1 --2- 172.21.15.88:0/2729654831 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbae80c38c0 0x7fbae8153090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:49.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.760+0000 7fbae6ffd700 1 -- 172.21.15.88:0/2729654831 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbae806a670 msgr2=0x7fbae814a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:49.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.760+0000 7fbae6ffd700 1 --2- 172.21.15.88:0/2729654831 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbae806a670 0x7fbae814a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:49.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.760+0000 7fbae6ffd700 1 -- 172.21.15.88:0/2729654831 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbae80c38c0 msgr2=0x7fbae8153090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:49.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.760+0000 7fbae6ffd700 1 --2- 172.21.15.88:0/2729654831 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbae80c38c0 0x7fbae8153090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:49.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.760+0000 7fbae6ffd700 1 -- 172.21.15.88:0/2729654831 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbae000b610 con 0x7fbae80c0b70 2023-10-18T16:07:49.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.760+0000 7fbae6ffd700 1 --2- 172.21.15.88:0/2729654831 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbae80c0b70 0x7fbae81455b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fbadc00c4c0 tx=0x7fbadc007de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:49.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.760+0000 7fbacf7fe700 1 -- 172.21.15.88:0/2729654831 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbadc00ce60 con 0x7fbae80c0b70 2023-10-18T16:07:49.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.760+0000 7fbacf7fe700 1 -- 172.21.15.88:0/2729654831 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbadc0072e0 con 0x7fbae80c0b70 2023-10-18T16:07:49.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.760+0000 7fbacf7fe700 1 -- 172.21.15.88:0/2729654831 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbadc0056e0 con 0x7fbae80c0b70 2023-10-18T16:07:49.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.760+0000 7fbaedc9d700 1 -- 172.21.15.88:0/2729654831 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbae81489d0 con 0x7fbae80c0b70 2023-10-18T16:07:49.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.760+0000 7fbaedc9d700 1 -- 172.21.15.88:0/2729654831 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbae815f210 con 0x7fbae80c0b70 2023-10-18T16:07:49.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.761+0000 7fbaedc9d700 1 -- 172.21.15.88:0/2729654831 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fbae806c4e0 con 0x7fbae80c0b70 2023-10-18T16:07:49.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.761+0000 7fbacf7fe700 1 -- 172.21.15.88:0/2729654831 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7fbadc005a90 con 0x7fbae80c0b70 2023-10-18T16:07:49.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.762+0000 7fbacf7fe700 1 --2- 172.21.15.88:0/2729654831 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fbad405f150 0x7fbad4061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:49.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.762+0000 7fbacf7fe700 1 -- 172.21.15.88:0/2729654831 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6121+0+0 (secure 0 0 0) 0x7fbadc0798d0 con 0x7fbae80c0b70 2023-10-18T16:07:49.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.762+0000 7fbacf7fe700 1 --2- 172.21.15.88:0/2729654831 >> [v2:172.21.15.151:6816/91544,v1:172.21.15.151:6817/91544] conn(0x7fbad4068470 0x7fbad406a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:49.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.762+0000 7fbacf7fe700 1 -- 172.21.15.88:0/2729654831 --> [v2:172.21.15.151:6816/91544,v1:172.21.15.151:6817/91544] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbad406af50 con 0x7fbad4068470 2023-10-18T16:07:49.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.762+0000 7fbae77fe700 1 --2- 172.21.15.88:0/2729654831 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fbad405f150 0x7fbad4061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:49.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.762+0000 7fbacf7fe700 1 -- 172.21.15.88:0/2729654831 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbadc07e220 con 0x7fbae80c0b70 2023-10-18T16:07:49.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.762+0000 7fbae7fff700 1 --2- 172.21.15.88:0/2729654831 >> [v2:172.21.15.151:6816/91544,v1:172.21.15.151:6817/91544] conn(0x7fbad4068470 0x7fbad406a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:49.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.762+0000 7fbae77fe700 1 --2- 172.21.15.88:0/2729654831 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fbad405f150 0x7fbad4061610 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fbad800f570 tx=0x7fbad8007730).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:49.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.762+0000 7fbae7fff700 1 --2- 172.21.15.88:0/2729654831 >> [v2:172.21.15.151:6816/91544,v1:172.21.15.151:6817/91544] conn(0x7fbad4068470 0x7fbad406a8d0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:49.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.763+0000 7fbacf7fe700 1 -- 172.21.15.88:0/2729654831 <== osd.6 v2:172.21.15.151:6816/91544 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23065 (crc 0 0 0) 0x7fbad406af50 con 0x7fbad4068470 2023-10-18T16:07:49.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.777+0000 7fbaedc9d700 1 -- 172.21.15.88:0/2729654831 --> [v2:172.21.15.151:6816/91544,v1:172.21.15.151:6817/91544] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fbae80bf6b0 con 0x7fbad4068470 2023-10-18T16:07:49.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.778+0000 7fbacf7fe700 1 -- 172.21.15.88:0/2729654831 <== osd.6 v2:172.21.15.151:6816/91544 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fbae80bf6b0 con 0x7fbad4068470 2023-10-18T16:07:49.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.778+0000 7fbaedc9d700 1 -- 172.21.15.88:0/2729654831 >> [v2:172.21.15.151:6816/91544,v1:172.21.15.151:6817/91544] conn(0x7fbad4068470 msgr2=0x7fbad406a8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:49.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.778+0000 7fbaedc9d700 1 --2- 172.21.15.88:0/2729654831 >> [v2:172.21.15.151:6816/91544,v1:172.21.15.151:6817/91544] conn(0x7fbad4068470 0x7fbad406a8d0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:49.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.778+0000 7fbaedc9d700 1 -- 172.21.15.88:0/2729654831 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fbad405f150 msgr2=0x7fbad4061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:49.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.778+0000 7fbaedc9d700 1 --2- 172.21.15.88:0/2729654831 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fbad405f150 0x7fbad4061610 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fbad800f570 tx=0x7fbad8007730).stop 2023-10-18T16:07:49.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.778+0000 7fbaedc9d700 1 -- 172.21.15.88:0/2729654831 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbae80c0b70 msgr2=0x7fbae81455b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:49.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.778+0000 7fbaedc9d700 1 --2- 172.21.15.88:0/2729654831 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbae80c0b70 0x7fbae81455b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fbadc00c4c0 tx=0x7fbadc007de0).stop 2023-10-18T16:07:49.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.778+0000 7fbae7fff700 1 -- 172.21.15.88:0/2729654831 reap_dead start 2023-10-18T16:07:49.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.778+0000 7fbaedc9d700 1 -- 172.21.15.88:0/2729654831 shutdown_connections 2023-10-18T16:07:49.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.778+0000 7fbaedc9d700 1 -- 172.21.15.88:0/2729654831 >> 172.21.15.88:0/2729654831 conn(0x7fbae8065cd0 msgr2=0x7fbae8151900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:49.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.778+0000 7fbaedc9d700 1 -- 172.21.15.88:0/2729654831 shutdown_connections 2023-10-18T16:07:49.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.778+0000 7fbaedc9d700 1 -- 172.21.15.88:0/2729654831 wait complete. 2023-10-18T16:07:49.793 INFO:teuthology.orchestra.run.smithi088.stdout:55834574854 2023-10-18T16:07:49.794 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-10-18T16:07:49.908 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.904+0000 7f8a1a70b700 1 Processor -- start 2023-10-18T16:07:49.908 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.904+0000 7f8a1a70b700 1 -- start start 2023-10-18T16:07:49.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.905+0000 7f8a1a70b700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8a140c20a0 0x7f8a14154280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:49.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.905+0000 7f8a1a70b700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8a14150340 0x7f8a141507f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:49.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.905+0000 7f8a1a70b700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a140a8570 con 0x7f8a1414d880 2023-10-18T16:07:49.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.905+0000 7f8a1a70b700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a14154770 con 0x7f8a14150340 2023-10-18T16:07:49.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.905+0000 7f8a1a70b700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a140a87c0 con 0x7f8a140c20a0 2023-10-18T16:07:49.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.905+0000 7f8a18ca8700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8a14150340 0x7f8a141507f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:49.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.905+0000 7f8a18ca8700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8a14150340 0x7f8a141507f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59612/0 (socket says 172.21.15.88:59612) 2023-10-18T16:07:49.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.905+0000 7f8a18ca8700 1 -- 172.21.15.88:0/2491191233 learned_addr learned my addr 172.21.15.88:0/2491191233 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:49.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.905+0000 7f8a137fe700 1 --2- 172.21.15.88:0/2491191233 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8a140c20a0 0x7f8a14154280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:49.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.905+0000 7f8a18ca8700 1 -- 172.21.15.88:0/2491191233 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8a140c20a0 msgr2=0x7f8a14154280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:49.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.906+0000 7f8a18ca8700 1 --2- 172.21.15.88:0/2491191233 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8a140c20a0 0x7f8a14154280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:49.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.906+0000 7f8a18ca8700 1 -- 172.21.15.88:0/2491191233 >> v1:172.21.15.151:6789/0 conn(0x7f8a1414d880 legacy=0x7f8a140c1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:49.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.906+0000 7f8a18ca8700 1 -- 172.21.15.88:0/2491191233 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a140a8aa0 con 0x7f8a14150340 2023-10-18T16:07:49.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.906+0000 7f8a18ca8700 1 --2- 172.21.15.88:0/2491191233 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a14150340 0x7f8a141507f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f8a0c009f20 tx=0x7f8a0c009840).ready entity=mon.0 client_cookie=2fbc998d4f187fab server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:49.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.906+0000 7f8a12ffd700 1 -- 172.21.15.88:0/2491191233 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a0c00e070 con 0x7f8a14150340 2023-10-18T16:07:49.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.906+0000 7f8a12ffd700 1 -- 172.21.15.88:0/2491191233 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a0c00b9d0 con 0x7f8a14150340 2023-10-18T16:07:49.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.906+0000 7f8a12ffd700 1 -- 172.21.15.88:0/2491191233 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a0c00bd50 con 0x7f8a14150340 2023-10-18T16:07:49.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.906+0000 7f8a1a70b700 1 -- 172.21.15.88:0/2491191233 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a14150340 msgr2=0x7f8a141507f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:49.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.906+0000 7f8a1a70b700 1 --2- 172.21.15.88:0/2491191233 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a14150340 0x7f8a141507f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f8a0c009f20 tx=0x7f8a0c009840).stop 2023-10-18T16:07:49.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.906+0000 7f8a1a70b700 1 -- 172.21.15.88:0/2491191233 shutdown_connections 2023-10-18T16:07:49.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.906+0000 7f8a1a70b700 1 --2- 172.21.15.88:0/2491191233 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a14150340 0x7f8a141507f0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:49.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.906+0000 7f8a1a70b700 1 --2- 172.21.15.88:0/2491191233 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8a140c20a0 0x7f8a14154280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:49.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.906+0000 7f8a1a70b700 1 -- 172.21.15.88:0/2491191233 >> 172.21.15.88:0/2491191233 conn(0x7f8a14069d80 msgr2=0x7f8a1406a190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:49.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.906+0000 7f8a1a70b700 1 -- 172.21.15.88:0/2491191233 shutdown_connections 2023-10-18T16:07:49.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.906+0000 7f8a1a70b700 1 -- 172.21.15.88:0/2491191233 wait complete. 2023-10-18T16:07:49.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.907+0000 7f8a1a70b700 1 Processor -- start 2023-10-18T16:07:49.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.908+0000 7f8a1a70b700 1 -- start start 2023-10-18T16:07:49.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.908+0000 7f8a1a70b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a140c20a0 0x7f8a140c3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:49.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.908+0000 7f8a1a70b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8a1414d880 0x7f8a140ba790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:49.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.908+0000 7f8a1a70b700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8a14150340 0x7f8a140bac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:49.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.908+0000 7f8a1a70b700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a140bca70 con 0x7f8a140c20a0 2023-10-18T16:07:49.923 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.908+0000 7f8a1a70b700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a140ac210 con 0x7f8a1414d880 2023-10-18T16:07:49.923 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.908+0000 7f8a1a70b700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a14153c60 con 0x7f8a14150340 2023-10-18T16:07:49.923 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.908+0000 7f8a13fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a140c20a0 0x7f8a140c3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:49.923 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.908+0000 7f8a13fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a140c20a0 0x7f8a140c3970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59622/0 (socket says 172.21.15.88:59622) 2023-10-18T16:07:49.924 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.908+0000 7f8a13fff700 1 -- 172.21.15.88:0/4184905641 learned_addr learned my addr 172.21.15.88:0/4184905641 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:49.924 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.909+0000 7f8a13fff700 1 -- 172.21.15.88:0/4184905641 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8a14150340 msgr2=0x7f8a140bac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:49.924 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.909+0000 7f8a18ca8700 1 --2- 172.21.15.88:0/4184905641 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8a14150340 0x7f8a140bac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:49.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.909+0000 7f8a137fe700 1 --2- 172.21.15.88:0/4184905641 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8a1414d880 0x7f8a140ba790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:49.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.909+0000 7f8a13fff700 1 --2- 172.21.15.88:0/4184905641 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8a14150340 0x7f8a140bac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:49.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.909+0000 7f8a13fff700 1 -- 172.21.15.88:0/4184905641 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8a1414d880 msgr2=0x7f8a140ba790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:49.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.909+0000 7f8a13fff700 1 --2- 172.21.15.88:0/4184905641 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8a1414d880 0x7f8a140ba790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:49.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.909+0000 7f8a13fff700 1 -- 172.21.15.88:0/4184905641 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a0c009ae0 con 0x7f8a140c20a0 2023-10-18T16:07:49.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.909+0000 7f8a13fff700 1 --2- 172.21.15.88:0/4184905641 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a140c20a0 0x7f8a140c3970 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f8a0400c4c0 tx=0x7f8a04007e00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:49.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.909+0000 7f89fbfff700 1 -- 172.21.15.88:0/4184905641 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a04007020 con 0x7f8a140c20a0 2023-10-18T16:07:49.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.909+0000 7f89fbfff700 1 -- 172.21.15.88:0/4184905641 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a04001e90 con 0x7f8a140c20a0 2023-10-18T16:07:49.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.909+0000 7f8a1a70b700 1 -- 172.21.15.88:0/4184905641 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a140bccc0 con 0x7f8a140c20a0 2023-10-18T16:07:49.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.909+0000 7f89fbfff700 1 -- 172.21.15.88:0/4184905641 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a040073b0 con 0x7f8a140c20a0 2023-10-18T16:07:49.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.909+0000 7f8a1a70b700 1 -- 172.21.15.88:0/4184905641 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a140ac4d0 con 0x7f8a140c20a0 2023-10-18T16:07:49.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.910+0000 7f8a1a70b700 1 -- 172.21.15.88:0/4184905641 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8a1406c4e0 con 0x7f8a140c20a0 2023-10-18T16:07:49.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.910+0000 7f89fbfff700 1 -- 172.21.15.88:0/4184905641 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f8a040068b0 con 0x7f8a140c20a0 2023-10-18T16:07:49.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.910+0000 7f89fbfff700 1 --2- 172.21.15.88:0/4184905641 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f8a0005f180 0x7f8a00061640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:49.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.911+0000 7f89fbfff700 1 -- 172.21.15.88:0/4184905641 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6121+0+0 (secure 0 0 0) 0x7f8a04079bf0 con 0x7f8a140c20a0 2023-10-18T16:07:49.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.911+0000 7f8a137fe700 1 --2- 172.21.15.88:0/4184905641 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f8a0005f180 0x7f8a00061640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:49.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.911+0000 7f89fbfff700 1 --2- 172.21.15.88:0/4184905641 >> [v2:172.21.15.151:6824/91545,v1:172.21.15.151:6825/91545] conn(0x7f8a000684a0 0x7f8a0006a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:49.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.911+0000 7f89fbfff700 1 -- 172.21.15.88:0/4184905641 --> [v2:172.21.15.151:6824/91545,v1:172.21.15.151:6825/91545] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8a0006af80 con 0x7f8a000684a0 2023-10-18T16:07:49.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.911+0000 7f89fbfff700 1 -- 172.21.15.88:0/4184905641 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8a0407c210 con 0x7f8a140c20a0 2023-10-18T16:07:49.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.911+0000 7f8a18ca8700 1 --2- 172.21.15.88:0/4184905641 >> [v2:172.21.15.151:6824/91545,v1:172.21.15.151:6825/91545] conn(0x7f8a000684a0 0x7f8a0006a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:49.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.911+0000 7f8a137fe700 1 --2- 172.21.15.88:0/4184905641 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f8a0005f180 0x7f8a00061640 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f8a08009270 tx=0x7f8a08008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:49.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.911+0000 7f8a18ca8700 1 --2- 172.21.15.88:0/4184905641 >> [v2:172.21.15.151:6824/91545,v1:172.21.15.151:6825/91545] conn(0x7f8a000684a0 0x7f8a0006a900 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:49.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.912+0000 7f89fbfff700 1 -- 172.21.15.88:0/4184905641 <== osd.7 v2:172.21.15.151:6824/91545 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23065 (crc 0 0 0) 0x7f8a0006af80 con 0x7f8a000684a0 2023-10-18T16:07:49.931 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.926+0000 7f8a1a70b700 1 -- 172.21.15.88:0/4184905641 --> [v2:172.21.15.151:6824/91545,v1:172.21.15.151:6825/91545] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8a140a8d80 con 0x7f8a000684a0 2023-10-18T16:07:49.931 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.926+0000 7f89fbfff700 1 -- 172.21.15.88:0/4184905641 <== osd.7 v2:172.21.15.151:6824/91545 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f8a140a8d80 con 0x7f8a000684a0 2023-10-18T16:07:49.932 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.927+0000 7f8a1a70b700 1 -- 172.21.15.88:0/4184905641 >> [v2:172.21.15.151:6824/91545,v1:172.21.15.151:6825/91545] conn(0x7f8a000684a0 msgr2=0x7f8a0006a900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:49.933 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.927+0000 7f8a1a70b700 1 --2- 172.21.15.88:0/4184905641 >> [v2:172.21.15.151:6824/91545,v1:172.21.15.151:6825/91545] conn(0x7f8a000684a0 0x7f8a0006a900 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:49.933 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.927+0000 7f8a1a70b700 1 -- 172.21.15.88:0/4184905641 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f8a0005f180 msgr2=0x7f8a00061640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:49.933 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.927+0000 7f8a1a70b700 1 --2- 172.21.15.88:0/4184905641 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f8a0005f180 0x7f8a00061640 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f8a08009270 tx=0x7f8a08008040).stop 2023-10-18T16:07:49.933 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.927+0000 7f8a1a70b700 1 -- 172.21.15.88:0/4184905641 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a140c20a0 msgr2=0x7f8a140c3970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:49.933 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.927+0000 7f8a1a70b700 1 --2- 172.21.15.88:0/4184905641 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a140c20a0 0x7f8a140c3970 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f8a0400c4c0 tx=0x7f8a04007e00).stop 2023-10-18T16:07:49.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.927+0000 7f8a18ca8700 1 -- 172.21.15.88:0/4184905641 reap_dead start 2023-10-18T16:07:49.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.927+0000 7f8a1a70b700 1 -- 172.21.15.88:0/4184905641 shutdown_connections 2023-10-18T16:07:49.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.927+0000 7f8a1a70b700 1 -- 172.21.15.88:0/4184905641 >> 172.21.15.88:0/4184905641 conn(0x7f8a14069d80 msgr2=0x7f8a1414c160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:49.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.927+0000 7f8a1a70b700 1 -- 172.21.15.88:0/4184905641 shutdown_connections 2023-10-18T16:07:49.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:49.927+0000 7f8a1a70b700 1 -- 172.21.15.88:0/4184905641 wait complete. 2023-10-18T16:07:49.942 INFO:teuthology.orchestra.run.smithi088.stdout:55834574853 2023-10-18T16:07:49.943 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-18T16:07:50.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.054+0000 7fe378b55700 1 Processor -- start 2023-10-18T16:07:50.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.054+0000 7fe378b55700 1 -- start start 2023-10-18T16:07:50.061 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.055+0000 7fe378b55700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe37414d8f0 0x7fe3740c2e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:50.061 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.055+0000 7fe378b55700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fe3740a9080 0x7fe3741503b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:50.061 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.055+0000 7fe378b55700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe3741508a0 0x7fe3740c1f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:50.062 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.055+0000 7fe378b55700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3740a7190 con 0x7fe37414d8f0 2023-10-18T16:07:50.062 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.055+0000 7fe378b55700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3740c3360 con 0x7fe3740a9080 2023-10-18T16:07:50.062 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.055+0000 7fe378b55700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe374150d30 con 0x7fe3741508a0 2023-10-18T16:07:50.062 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.055+0000 7fe37259c700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe37414d8f0 0x7fe3740c2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:50.063 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.055+0000 7fe37259c700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe37414d8f0 0x7fe3740c2e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59630/0 (socket says 172.21.15.88:59630) 2023-10-18T16:07:50.063 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.055+0000 7fe37259c700 1 -- 172.21.15.88:0/2015538381 learned_addr learned my addr 172.21.15.88:0/2015538381 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:50.063 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.055+0000 7fe37259c700 1 -- 172.21.15.88:0/2015538381 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe3741508a0 msgr2=0x7fe3740c1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:50.063 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.055+0000 7fe37259c700 1 --2- 172.21.15.88:0/2015538381 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe3741508a0 0x7fe3740c1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:07:50.063 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.055+0000 7fe37259c700 1 -- 172.21.15.88:0/2015538381 >> v2:172.21.15.151:3300/0 conn(0x7fe3740a9080 msgr2=0x7fe3741503b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:50.064 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.055+0000 7fe37259c700 1 --2- 172.21.15.88:0/2015538381 >> v2:172.21.15.151:3300/0 conn(0x7fe3740a9080 0x7fe3741503b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:07:50.064 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.055+0000 7fe37259c700 1 -- 172.21.15.88:0/2015538381 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3740c3e10 con 0x7fe37414d8f0 2023-10-18T16:07:50.064 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.055+0000 7fe37259c700 1 --2- 172.21.15.88:0/2015538381 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe37414d8f0 0x7fe3740c2e70 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7fe374084a50 tx=0x7fe3640090f0).ready entity=mon.0 client_cookie=7bfe6edc8a48a8ab server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:50.064 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.056+0000 7fe37159a700 1 -- 172.21.15.88:0/2015538381 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe36400b600 con 0x7fe37414d8f0 2023-10-18T16:07:50.064 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.056+0000 7fe37159a700 1 -- 172.21.15.88:0/2015538381 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe36400b760 con 0x7fe37414d8f0 2023-10-18T16:07:50.065 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.056+0000 7fe37159a700 1 -- 172.21.15.88:0/2015538381 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe36400bae0 con 0x7fe37414d8f0 2023-10-18T16:07:50.065 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.056+0000 7fe378b55700 1 -- 172.21.15.88:0/2015538381 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe37414d8f0 msgr2=0x7fe3740c2e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:50.065 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.056+0000 7fe378b55700 1 --2- 172.21.15.88:0/2015538381 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe37414d8f0 0x7fe3740c2e70 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7fe374084a50 tx=0x7fe3640090f0).stop 2023-10-18T16:07:50.065 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.056+0000 7fe378b55700 1 -- 172.21.15.88:0/2015538381 shutdown_connections 2023-10-18T16:07:50.065 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.056+0000 7fe378b55700 1 --2- 172.21.15.88:0/2015538381 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe3741508a0 0x7fe3740c1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:07:50.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.056+0000 7fe378b55700 1 --2- 172.21.15.88:0/2015538381 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe37414d8f0 0x7fe3740c2e70 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:50.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.056+0000 7fe378b55700 1 --2- 172.21.15.88:0/2015538381 >> v2:172.21.15.151:3300/0 conn(0x7fe3740a9080 0x7fe3741503b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:07:50.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.056+0000 7fe378b55700 1 -- 172.21.15.88:0/2015538381 >> 172.21.15.88:0/2015538381 conn(0x7fe374069d80 msgr2=0x7fe37406a190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:50.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.056+0000 7fe378b55700 1 -- 172.21.15.88:0/2015538381 shutdown_connections 2023-10-18T16:07:50.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.056+0000 7fe378b55700 1 -- 172.21.15.88:0/2015538381 wait complete. 2023-10-18T16:07:50.067 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.057+0000 7fe378b55700 1 Processor -- start 2023-10-18T16:07:50.067 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.057+0000 7fe378b55700 1 -- start start 2023-10-18T16:07:50.070 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.058+0000 7fe378b55700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe3740a9080 0x7fe37414f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:50.070 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.058+0000 7fe378b55700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe37414d8f0 0x7fe3740aabf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:50.071 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.058+0000 7fe378b55700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3741508a0 0x7fe374160560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:50.071 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.058+0000 7fe378b55700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3740c6260 con 0x7fe3741508a0 2023-10-18T16:07:50.071 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.058+0000 7fe378b55700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe37414fe80 con 0x7fe3740a9080 2023-10-18T16:07:50.072 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.058+0000 7fe378b55700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3740c6440 con 0x7fe37414d8f0 2023-10-18T16:07:50.072 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.058+0000 7fe372d9d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3741508a0 0x7fe374160560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:50.072 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.058+0000 7fe372d9d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3741508a0 0x7fe374160560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59634/0 (socket says 172.21.15.88:59634) 2023-10-18T16:07:50.072 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.058+0000 7fe372d9d700 1 -- 172.21.15.88:0/3087825524 learned_addr learned my addr 172.21.15.88:0/3087825524 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:50.073 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.058+0000 7fe37259c700 1 --2- 172.21.15.88:0/3087825524 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe3740a9080 0x7fe37414f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:50.073 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.058+0000 7fe371d9b700 1 --2- 172.21.15.88:0/3087825524 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe37414d8f0 0x7fe3740aabf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:50.073 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.058+0000 7fe372d9d700 1 -- 172.21.15.88:0/3087825524 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe37414d8f0 msgr2=0x7fe3740aabf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:50.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.058+0000 7fe372d9d700 1 --2- 172.21.15.88:0/3087825524 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe37414d8f0 0x7fe3740aabf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:50.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.058+0000 7fe372d9d700 1 -- 172.21.15.88:0/3087825524 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe3740a9080 msgr2=0x7fe37414f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:50.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.058+0000 7fe372d9d700 1 --2- 172.21.15.88:0/3087825524 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe3740a9080 0x7fe37414f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:50.075 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.058+0000 7fe372d9d700 1 -- 172.21.15.88:0/3087825524 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe364009370 con 0x7fe3741508a0 2023-10-18T16:07:50.075 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.058+0000 7fe37259c700 1 --2- 172.21.15.88:0/3087825524 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe3740a9080 0x7fe37414f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:07:50.075 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.058+0000 7fe372d9d700 1 --2- 172.21.15.88:0/3087825524 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3741508a0 0x7fe374160560 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fe36c00b8a0 tx=0x7fe36c0093e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:50.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.059+0000 7fe3627fc700 1 -- 172.21.15.88:0/3087825524 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe36c00c810 con 0x7fe3741508a0 2023-10-18T16:07:50.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.059+0000 7fe3627fc700 1 -- 172.21.15.88:0/3087825524 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe36c00c040 con 0x7fe3741508a0 2023-10-18T16:07:50.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.059+0000 7fe3627fc700 1 -- 172.21.15.88:0/3087825524 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe36c00c970 con 0x7fe3741508a0 2023-10-18T16:07:50.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.059+0000 7fe378b55700 1 -- 172.21.15.88:0/3087825524 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3740c6720 con 0x7fe3741508a0 2023-10-18T16:07:50.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.059+0000 7fe378b55700 1 -- 172.21.15.88:0/3087825524 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3740c6ce0 con 0x7fe3741508a0 2023-10-18T16:07:50.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.060+0000 7fe378b55700 1 -- 172.21.15.88:0/3087825524 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe37406c4e0 con 0x7fe3741508a0 2023-10-18T16:07:50.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.060+0000 7fe3627fc700 1 -- 172.21.15.88:0/3087825524 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7fe36c012020 con 0x7fe3741508a0 2023-10-18T16:07:50.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.060+0000 7fe3627fc700 1 --2- 172.21.15.88:0/3087825524 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fe35405f150 0x7fe354061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:50.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.060+0000 7fe3627fc700 1 -- 172.21.15.88:0/3087825524 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6121+0+0 (secure 0 0 0) 0x7fe36c0794c0 con 0x7fe3741508a0 2023-10-18T16:07:50.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.060+0000 7fe37259c700 1 --2- 172.21.15.88:0/3087825524 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fe35405f150 0x7fe354061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:50.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.061+0000 7fe37259c700 1 --2- 172.21.15.88:0/3087825524 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fe35405f150 0x7fe354061610 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7fe364009130 tx=0x7fe364017000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:50.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.064+0000 7fe3627fc700 1 -- 172.21.15.88:0/3087825524 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7fe36c04a730 con 0x7fe3741508a0 2023-10-18T16:07:50.230 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.226+0000 7fe378b55700 1 -- 172.21.15.88:0/3087825524 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fe3740c2c50 con 0x7fe3741508a0 2023-10-18T16:07:50.230 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.226+0000 7fe3627fc700 1 -- 172.21.15.88:0/3087825524 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe36c01a070 con 0x7fe3741508a0 2023-10-18T16:07:50.231 INFO:teuthology.orchestra.run.smithi088.stdout:55834574851 2023-10-18T16:07:50.232 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.228+0000 7fe378b55700 1 -- 172.21.15.88:0/3087825524 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fe35405f150 msgr2=0x7fe354061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:50.232 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.228+0000 7fe378b55700 1 --2- 172.21.15.88:0/3087825524 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fe35405f150 0x7fe354061610 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7fe364009130 tx=0x7fe364017000).stop 2023-10-18T16:07:50.232 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.228+0000 7fe378b55700 1 -- 172.21.15.88:0/3087825524 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3741508a0 msgr2=0x7fe374160560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:50.233 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.228+0000 7fe378b55700 1 --2- 172.21.15.88:0/3087825524 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3741508a0 0x7fe374160560 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fe36c00b8a0 tx=0x7fe36c0093e0).stop 2023-10-18T16:07:50.233 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.228+0000 7fe378b55700 1 -- 172.21.15.88:0/3087825524 shutdown_connections 2023-10-18T16:07:50.233 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.228+0000 7fe378b55700 1 --2- 172.21.15.88:0/3087825524 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fe35405f150 0x7fe354061610 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:50.233 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.228+0000 7fe378b55700 1 --2- 172.21.15.88:0/3087825524 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3741508a0 0x7fe374160560 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:50.233 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.228+0000 7fe378b55700 1 --2- 172.21.15.88:0/3087825524 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe3740a9080 0x7fe37414f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:50.234 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.228+0000 7fe378b55700 1 --2- 172.21.15.88:0/3087825524 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe37414d8f0 0x7fe3740aabf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:50.234 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.228+0000 7fe378b55700 1 -- 172.21.15.88:0/3087825524 >> 172.21.15.88:0/3087825524 conn(0x7fe374069d80 msgr2=0x7fe3740c1f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:50.234 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.228+0000 7fe378b55700 1 -- 172.21.15.88:0/3087825524 shutdown_connections 2023-10-18T16:07:50.234 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:50.228+0000 7fe378b55700 1 -- 172.21.15.88:0/3087825524 wait complete. 2023-10-18T16:07:50.243 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574851 for osd.0 2023-10-18T16:07:51.245 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-18T16:07:51.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.368+0000 7f8651026700 1 Processor -- start 2023-10-18T16:07:51.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.368+0000 7f8651026700 1 -- start start 2023-10-18T16:07:51.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.369+0000 7f8651026700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f864c14b150 0x7f864c14e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:51.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.369+0000 7f8651026700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f864c067110 0x7f864c0675a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:51.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.369+0000 7f8651026700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f864c0691f0 con 0x7f864c14b530 2023-10-18T16:07:51.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.369+0000 7f8651026700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f864c0685e0 con 0x7f864c067110 2023-10-18T16:07:51.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.369+0000 7f8651026700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f864c0688c0 con 0x7f864c14b150 2023-10-18T16:07:51.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.370+0000 7f8650825700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f864c067110 0x7f864c0675a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:51.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.370+0000 7f864bfff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f864c14b150 0x7f864c14e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:51.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.370+0000 7f8650825700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f864c067110 0x7f864c0675a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.88:60182/0 (socket says 172.21.15.88:60182) 2023-10-18T16:07:51.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.370+0000 7f8650825700 1 -- 172.21.15.88:0/3098207353 learned_addr learned my addr 172.21.15.88:0/3098207353 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:51.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.370+0000 7f864bfff700 1 -- 172.21.15.88:0/3098207353 >> v2:172.21.15.151:3300/0 conn(0x7f864c067110 msgr2=0x7f864c0675a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:51.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.370+0000 7f864bfff700 1 --2- 172.21.15.88:0/3098207353 >> v2:172.21.15.151:3300/0 conn(0x7f864c067110 0x7f864c0675a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:51.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.370+0000 7f864bfff700 1 -- 172.21.15.88:0/3098207353 >> v1:172.21.15.151:6789/0 conn(0x7f864c14b530 legacy=0x7f864c0b9670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:51.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.370+0000 7f864bfff700 1 -- 172.21.15.88:0/3098207353 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f864c068ba0 con 0x7f864c14b150 2023-10-18T16:07:51.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.370+0000 7f8650825700 1 --2- 172.21.15.88:0/3098207353 >> v2:172.21.15.151:3300/0 conn(0x7f864c067110 0x7f864c0675a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2023-10-18T16:07:51.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.370+0000 7f864bfff700 1 --2- 172.21.15.88:0/3098207353 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f864c14b150 0x7f864c14e810 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f863c009760 tx=0x7f863c009060).ready entity=mon.2 client_cookie=1740c8515f3bf590 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:51.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.370+0000 7f864affd700 1 -- 172.21.15.88:0/3098207353 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f863c00f510 con 0x7f864c14b150 2023-10-18T16:07:51.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.370+0000 7f864affd700 1 -- 172.21.15.88:0/3098207353 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f863c00f670 con 0x7f864c14b150 2023-10-18T16:07:51.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.370+0000 7f864affd700 1 -- 172.21.15.88:0/3098207353 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f863c00f9f0 con 0x7f864c14b150 2023-10-18T16:07:51.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.371+0000 7f8651026700 1 -- 172.21.15.88:0/3098207353 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f864c14b150 msgr2=0x7f864c14e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:51.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.371+0000 7f8651026700 1 --2- 172.21.15.88:0/3098207353 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f864c14b150 0x7f864c14e810 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f863c009760 tx=0x7f863c009060).stop 2023-10-18T16:07:51.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.371+0000 7f8651026700 1 -- 172.21.15.88:0/3098207353 shutdown_connections 2023-10-18T16:07:51.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.371+0000 7f8651026700 1 --2- 172.21.15.88:0/3098207353 >> v2:172.21.15.151:3300/0 conn(0x7f864c067110 0x7f864c0675a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:51.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.371+0000 7f8651026700 1 --2- 172.21.15.88:0/3098207353 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f864c14b150 0x7f864c14e810 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:51.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.371+0000 7f8651026700 1 -- 172.21.15.88:0/3098207353 >> 172.21.15.88:0/3098207353 conn(0x7f864c062650 msgr2=0x7f864c062a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:51.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.371+0000 7f8651026700 1 -- 172.21.15.88:0/3098207353 shutdown_connections 2023-10-18T16:07:51.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.371+0000 7f8651026700 1 -- 172.21.15.88:0/3098207353 wait complete. 2023-10-18T16:07:51.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.372+0000 7f8651026700 1 Processor -- start 2023-10-18T16:07:51.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.372+0000 7f8651026700 1 -- start start 2023-10-18T16:07:51.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.373+0000 7f8651026700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f864c067110 0x7f864c145920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:51.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.373+0000 7f8651026700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f864c14b150 0x7f864c14dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:51.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.373+0000 7f8651026700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f864c145e10 0x7f864c14e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:51.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.373+0000 7f8651026700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f864c14e9d0 con 0x7f864c14b150 2023-10-18T16:07:51.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.373+0000 7f8651026700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f864c14daf0 con 0x7f864c145e10 2023-10-18T16:07:51.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.373+0000 7f8651026700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f864c14bbf0 con 0x7f864c067110 2023-10-18T16:07:51.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.373+0000 7f864b7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f864c14b150 0x7f864c14dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:51.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.373+0000 7f864b7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f864c14b150 0x7f864c14dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59636/0 (socket says 172.21.15.88:59636) 2023-10-18T16:07:51.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.373+0000 7f864b7fe700 1 -- 172.21.15.88:0/3510132522 learned_addr learned my addr 172.21.15.88:0/3510132522 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:51.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.373+0000 7f864bfff700 1 --2- 172.21.15.88:0/3510132522 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f864c067110 0x7f864c145920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:51.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.373+0000 7f8650825700 1 --2- 172.21.15.88:0/3510132522 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f864c145e10 0x7f864c14e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:51.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.373+0000 7f864b7fe700 1 -- 172.21.15.88:0/3510132522 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f864c067110 msgr2=0x7f864c145920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:51.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.373+0000 7f864b7fe700 1 --2- 172.21.15.88:0/3510132522 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f864c067110 0x7f864c145920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:51.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.373+0000 7f864b7fe700 1 -- 172.21.15.88:0/3510132522 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f864c145e10 msgr2=0x7f864c14e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:51.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.373+0000 7f864b7fe700 1 --2- 172.21.15.88:0/3510132522 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f864c145e10 0x7f864c14e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:51.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.373+0000 7f864b7fe700 1 -- 172.21.15.88:0/3510132522 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8644009ae0 con 0x7f864c14b150 2023-10-18T16:07:51.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.373+0000 7f8650825700 1 --2- 172.21.15.88:0/3510132522 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f864c145e10 0x7f864c14e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:07:51.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.374+0000 7f864b7fe700 1 --2- 172.21.15.88:0/3510132522 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f864c14b150 0x7f864c14dff0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f864000c440 tx=0x7f8640007d40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:51.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.374+0000 7f8633fff700 1 -- 172.21.15.88:0/3510132522 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8640007020 con 0x7f864c14b150 2023-10-18T16:07:51.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.374+0000 7f8633fff700 1 -- 172.21.15.88:0/3510132522 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8640007180 con 0x7f864c14b150 2023-10-18T16:07:51.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.374+0000 7f8651026700 1 -- 172.21.15.88:0/3510132522 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f863c0092a0 con 0x7f864c14b150 2023-10-18T16:07:51.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.374+0000 7f8633fff700 1 -- 172.21.15.88:0/3510132522 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8640007520 con 0x7f864c14b150 2023-10-18T16:07:51.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.374+0000 7f8651026700 1 -- 172.21.15.88:0/3510132522 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f864c14c1b0 con 0x7f864c14b150 2023-10-18T16:07:51.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.375+0000 7f8651026700 1 -- 172.21.15.88:0/3510132522 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f864c14c3d0 con 0x7f864c14b150 2023-10-18T16:07:51.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.375+0000 7f8633fff700 1 -- 172.21.15.88:0/3510132522 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f8640006ac0 con 0x7f864c14b150 2023-10-18T16:07:51.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.375+0000 7f8633fff700 1 --2- 172.21.15.88:0/3510132522 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f863805f150 0x7f8638061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:51.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.376+0000 7f8633fff700 1 -- 172.21.15.88:0/3510132522 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6121+0+0 (secure 0 0 0) 0x7f8640079e00 con 0x7f864c14b150 2023-10-18T16:07:51.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.376+0000 7f864bfff700 1 --2- 172.21.15.88:0/3510132522 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f863805f150 0x7f8638061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:51.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.376+0000 7f864bfff700 1 --2- 172.21.15.88:0/3510132522 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f863805f150 0x7f8638061610 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f863c009fd0 tx=0x7f863c017000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:51.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.379+0000 7f8633fff700 1 -- 172.21.15.88:0/3510132522 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f864004b0d0 con 0x7f864c14b150 2023-10-18T16:07:51.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.545+0000 7f8651026700 1 -- 172.21.15.88:0/3510132522 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f864c046c20 con 0x7f864c14b150 2023-10-18T16:07:51.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.545+0000 7f8633fff700 1 -- 172.21.15.88:0/3510132522 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f864004e6f0 con 0x7f864c14b150 2023-10-18T16:07:51.549 INFO:teuthology.orchestra.run.smithi088.stdout:55834574853 2023-10-18T16:07:51.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.547+0000 7f8651026700 1 -- 172.21.15.88:0/3510132522 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f863805f150 msgr2=0x7f8638061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:51.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.547+0000 7f8651026700 1 --2- 172.21.15.88:0/3510132522 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f863805f150 0x7f8638061610 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f863c009fd0 tx=0x7f863c017000).stop 2023-10-18T16:07:51.554 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.547+0000 7f8651026700 1 -- 172.21.15.88:0/3510132522 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f864c14b150 msgr2=0x7f864c14dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:51.554 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.547+0000 7f8651026700 1 --2- 172.21.15.88:0/3510132522 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f864c14b150 0x7f864c14dff0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f864000c440 tx=0x7f8640007d40).stop 2023-10-18T16:07:51.554 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.547+0000 7f8651026700 1 -- 172.21.15.88:0/3510132522 shutdown_connections 2023-10-18T16:07:51.554 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.547+0000 7f8651026700 1 --2- 172.21.15.88:0/3510132522 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f863805f150 0x7f8638061610 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:51.555 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.547+0000 7f8651026700 1 --2- 172.21.15.88:0/3510132522 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f864c145e10 0x7f864c14e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:51.555 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.547+0000 7f8651026700 1 --2- 172.21.15.88:0/3510132522 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f864c067110 0x7f864c145920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:51.555 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.547+0000 7f8651026700 1 --2- 172.21.15.88:0/3510132522 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f864c14b150 0x7f864c14dff0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:51.555 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.547+0000 7f8651026700 1 -- 172.21.15.88:0/3510132522 >> 172.21.15.88:0/3510132522 conn(0x7f864c062650 msgr2=0x7f864c0b9670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:51.555 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.547+0000 7f8651026700 1 -- 172.21.15.88:0/3510132522 shutdown_connections 2023-10-18T16:07:51.556 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.547+0000 7f8651026700 1 -- 172.21.15.88:0/3510132522 wait complete. 2023-10-18T16:07:51.563 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.0 2023-10-18T16:07:51.563 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-18T16:07:51.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.674+0000 7fd72255f700 1 Processor -- start 2023-10-18T16:07:51.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.674+0000 7fd72255f700 1 -- start start 2023-10-18T16:07:51.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.675+0000 7fd72255f700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd71c0be800 0x7fd71c0bc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:51.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.675+0000 7fd72255f700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fd71c0c0cf0 0x7fd71c14ec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:51.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.675+0000 7fd72255f700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd71c066e40 con 0x7fd71c0bbef0 2023-10-18T16:07:51.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.675+0000 7fd72255f700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd71c0bdc80 con 0x7fd71c0be800 2023-10-18T16:07:51.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.675+0000 7fd72255f700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd71c067100 con 0x7fd71c0c0cf0 2023-10-18T16:07:51.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.675+0000 7fd71b7fe700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd71c0be800 0x7fd71c0bc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:51.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.675+0000 7fd71bfff700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fd71c0bbef0 0x7fd71c0be0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:52510/0 (socket says 172.21.15.88:52510) 2023-10-18T16:07:51.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.675+0000 7fd71b7fe700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd71c0be800 0x7fd71c0bc2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59640/0 (socket says 172.21.15.88:59640) 2023-10-18T16:07:51.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.675+0000 7fd71bfff700 1 -- 172.21.15.88:0/3151953558 learned_addr learned my addr 172.21.15.88:0/3151953558 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:07:51.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.675+0000 7fd720afc700 1 --2- 172.21.15.88:0/3151953558 >> v2:172.21.15.151:3300/0 conn(0x7fd71c0c0cf0 0x7fd71c14ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:51.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.675+0000 7fd71b7fe700 1 -- 172.21.15.88:0/3151953558 >> v2:172.21.15.151:3300/0 conn(0x7fd71c0c0cf0 msgr2=0x7fd71c14ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:51.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.675+0000 7fd71b7fe700 1 --2- 172.21.15.88:0/3151953558 >> v2:172.21.15.151:3300/0 conn(0x7fd71c0c0cf0 0x7fd71c14ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:51.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.675+0000 7fd71b7fe700 1 -- 172.21.15.88:0/3151953558 >> v1:172.21.15.88:6789/0 conn(0x7fd71c0bbef0 legacy=0x7fd71c0be0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:51.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.675+0000 7fd71b7fe700 1 -- 172.21.15.88:0/3151953558 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd71c0673e0 con 0x7fd71c0be800 2023-10-18T16:07:51.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.675+0000 7fd71b7fe700 1 --2- 172.21.15.88:0/3151953558 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd71c0be800 0x7fd71c0bc2d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fd71c0b81c0 tx=0x7fd710009060).ready entity=mon.0 client_cookie=95c68adae474997b server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:51.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.675+0000 7fd71affd700 1 -- 172.21.15.88:0/3151953558 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd71000f510 con 0x7fd71c0be800 2023-10-18T16:07:51.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.675+0000 7fd71affd700 1 -- 172.21.15.88:0/3151953558 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd71000f670 con 0x7fd71c0be800 2023-10-18T16:07:51.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.675+0000 7fd71affd700 1 -- 172.21.15.88:0/3151953558 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd71000f9f0 con 0x7fd71c0be800 2023-10-18T16:07:51.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.676+0000 7fd72255f700 1 -- 172.21.15.88:0/3151953558 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd71c0be800 msgr2=0x7fd71c0bc2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:51.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.676+0000 7fd72255f700 1 --2- 172.21.15.88:0/3151953558 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd71c0be800 0x7fd71c0bc2d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fd71c0b81c0 tx=0x7fd710009060).stop 2023-10-18T16:07:51.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.676+0000 7fd72255f700 1 -- 172.21.15.88:0/3151953558 shutdown_connections 2023-10-18T16:07:51.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.676+0000 7fd72255f700 1 --2- 172.21.15.88:0/3151953558 >> v2:172.21.15.151:3300/0 conn(0x7fd71c0c0cf0 0x7fd71c14ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:51.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.676+0000 7fd72255f700 1 --2- 172.21.15.88:0/3151953558 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd71c0be800 0x7fd71c0bc2d0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:51.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.676+0000 7fd72255f700 1 -- 172.21.15.88:0/3151953558 >> 172.21.15.88:0/3151953558 conn(0x7fd71c0c48f0 msgr2=0x7fd71c0c4d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:51.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.676+0000 7fd72255f700 1 -- 172.21.15.88:0/3151953558 shutdown_connections 2023-10-18T16:07:51.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.676+0000 7fd72255f700 1 -- 172.21.15.88:0/3151953558 wait complete. 2023-10-18T16:07:51.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.677+0000 7fd72255f700 1 Processor -- start 2023-10-18T16:07:51.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.677+0000 7fd72255f700 1 -- start start 2023-10-18T16:07:51.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.678+0000 7fd72255f700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd71c0bbef0 0x7fd71c0b9650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:51.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.678+0000 7fd72255f700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd71c0be800 0x7fd71c14c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:51.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.678+0000 7fd72255f700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd71c0c0cf0 0x7fd71c147310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:51.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.678+0000 7fd72255f700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd71c0ba300 con 0x7fd71c0bbef0 2023-10-18T16:07:51.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.678+0000 7fd72255f700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd71c14db30 con 0x7fd71c0be800 2023-10-18T16:07:51.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.678+0000 7fd72255f700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd71c14de10 con 0x7fd71c0c0cf0 2023-10-18T16:07:51.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.678+0000 7fd71bfff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd71c0bbef0 0x7fd71c0b9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:51.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.678+0000 7fd71bfff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd71c0bbef0 0x7fd71c0b9650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59646/0 (socket says 172.21.15.88:59646) 2023-10-18T16:07:51.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.678+0000 7fd71bfff700 1 -- 172.21.15.88:0/1388748036 learned_addr learned my addr 172.21.15.88:0/1388748036 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:51.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.678+0000 7fd71b7fe700 1 --2- 172.21.15.88:0/1388748036 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd71c0be800 0x7fd71c14c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:51.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.678+0000 7fd720afc700 1 --2- 172.21.15.88:0/1388748036 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd71c0c0cf0 0x7fd71c147310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:51.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.678+0000 7fd71bfff700 1 -- 172.21.15.88:0/1388748036 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd71c0c0cf0 msgr2=0x7fd71c147310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:51.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.678+0000 7fd71bfff700 1 --2- 172.21.15.88:0/1388748036 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd71c0c0cf0 0x7fd71c147310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:51.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.678+0000 7fd71bfff700 1 -- 172.21.15.88:0/1388748036 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd71c0be800 msgr2=0x7fd71c14c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:51.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.678+0000 7fd71bfff700 1 --2- 172.21.15.88:0/1388748036 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd71c0be800 0x7fd71c14c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:51.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.678+0000 7fd71bfff700 1 -- 172.21.15.88:0/1388748036 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7100092a0 con 0x7fd71c0bbef0 2023-10-18T16:07:51.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.678+0000 7fd720afc700 1 --2- 172.21.15.88:0/1388748036 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd71c0c0cf0 0x7fd71c147310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:07:51.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.678+0000 7fd71bfff700 1 --2- 172.21.15.88:0/1388748036 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd71c0bbef0 0x7fd71c0b9650 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fd70c00c5e0 tx=0x7fd70c007db0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:51.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.678+0000 7fd703fff700 1 -- 172.21.15.88:0/1388748036 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd70c0072e0 con 0x7fd71c0bbef0 2023-10-18T16:07:51.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.678+0000 7fd703fff700 1 -- 172.21.15.88:0/1388748036 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd70c007440 con 0x7fd71c0bbef0 2023-10-18T16:07:51.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.678+0000 7fd72255f700 1 -- 172.21.15.88:0/1388748036 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd71c14e0f0 con 0x7fd71c0bbef0 2023-10-18T16:07:51.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.678+0000 7fd703fff700 1 -- 172.21.15.88:0/1388748036 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd70c0057a0 con 0x7fd71c0bbef0 2023-10-18T16:07:51.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.678+0000 7fd72255f700 1 -- 172.21.15.88:0/1388748036 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd71c0b9f90 con 0x7fd71c0bbef0 2023-10-18T16:07:51.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.679+0000 7fd72255f700 1 -- 172.21.15.88:0/1388748036 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd71c06c4e0 con 0x7fd71c0bbef0 2023-10-18T16:07:51.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.680+0000 7fd703fff700 1 -- 172.21.15.88:0/1388748036 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7fd70c005b50 con 0x7fd71c0bbef0 2023-10-18T16:07:51.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.680+0000 7fd703fff700 1 --2- 172.21.15.88:0/1388748036 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fd70805f150 0x7fd708061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:51.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.680+0000 7fd703fff700 1 -- 172.21.15.88:0/1388748036 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6121+0+0 (secure 0 0 0) 0x7fd70c0799d0 con 0x7fd71c0bbef0 2023-10-18T16:07:51.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.681+0000 7fd71b7fe700 1 --2- 172.21.15.88:0/1388748036 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fd70805f150 0x7fd708061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:51.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.681+0000 7fd71b7fe700 1 --2- 172.21.15.88:0/1388748036 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fd70805f150 0x7fd708061610 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fd71000a010 tx=0x7fd710017000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:51.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.684+0000 7fd703fff700 1 -- 172.21.15.88:0/1388748036 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7fd70c04ac40 con 0x7fd71c0bbef0 2023-10-18T16:07:51.851 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.848+0000 7fd72255f700 1 -- 172.21.15.88:0/1388748036 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fd71c05e360 con 0x7fd71c0bbef0 2023-10-18T16:07:51.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.848+0000 7fd703fff700 1 -- 172.21.15.88:0/1388748036 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd70c079020 con 0x7fd71c0bbef0 2023-10-18T16:07:51.852 INFO:teuthology.orchestra.run.smithi088.stdout:55834574853 2023-10-18T16:07:51.854 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.850+0000 7fd72255f700 1 -- 172.21.15.88:0/1388748036 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fd70805f150 msgr2=0x7fd708061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:51.854 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.850+0000 7fd72255f700 1 --2- 172.21.15.88:0/1388748036 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fd70805f150 0x7fd708061610 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fd71000a010 tx=0x7fd710017000).stop 2023-10-18T16:07:51.854 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.850+0000 7fd72255f700 1 -- 172.21.15.88:0/1388748036 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd71c0bbef0 msgr2=0x7fd71c0b9650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:51.854 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.850+0000 7fd72255f700 1 --2- 172.21.15.88:0/1388748036 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd71c0bbef0 0x7fd71c0b9650 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fd70c00c5e0 tx=0x7fd70c007db0).stop 2023-10-18T16:07:51.855 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.850+0000 7fd72255f700 1 -- 172.21.15.88:0/1388748036 shutdown_connections 2023-10-18T16:07:51.855 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.850+0000 7fd72255f700 1 --2- 172.21.15.88:0/1388748036 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fd70805f150 0x7fd708061610 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:51.855 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.850+0000 7fd72255f700 1 --2- 172.21.15.88:0/1388748036 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd71c0c0cf0 0x7fd71c147310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:51.855 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.850+0000 7fd72255f700 1 --2- 172.21.15.88:0/1388748036 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd71c0bbef0 0x7fd71c0b9650 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:51.856 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.850+0000 7fd72255f700 1 --2- 172.21.15.88:0/1388748036 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd71c0be800 0x7fd71c14c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:51.856 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.850+0000 7fd72255f700 1 -- 172.21.15.88:0/1388748036 >> 172.21.15.88:0/1388748036 conn(0x7fd71c0c48f0 msgr2=0x7fd71c0c1320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:51.856 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.850+0000 7fd72255f700 1 -- 172.21.15.88:0/1388748036 shutdown_connections 2023-10-18T16:07:51.856 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.850+0000 7fd72255f700 1 -- 172.21.15.88:0/1388748036 wait complete. 2023-10-18T16:07:51.865 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.1 2023-10-18T16:07:51.866 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-18T16:07:51.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.976+0000 7f6d52fcb700 1 Processor -- start 2023-10-18T16:07:51.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.976+0000 7f6d52fcb700 1 -- start start 2023-10-18T16:07:51.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.977+0000 7f6d52fcb700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6d4c0c20a0 0x7f6d4c154280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:51.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.977+0000 7f6d52fcb700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d4c0a8570 con 0x7f6d4c150340 2023-10-18T16:07:51.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.977+0000 7f6d52fcb700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d4c154770 con 0x7f6d4c14d880 2023-10-18T16:07:51.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.977+0000 7f6d52fcb700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d4c0a87c0 con 0x7f6d4c0c20a0 2023-10-18T16:07:51.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.977+0000 7f6d43fff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6d4c0c20a0 0x7f6d4c154280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:51.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.977+0000 7f6d43fff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6d4c0c20a0 0x7f6d4c154280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59648/0 (socket says 172.21.15.88:59648) 2023-10-18T16:07:51.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.977+0000 7f6d51568700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f6d4c150340 0x7f6d4c1507d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:52526/0 (socket says 172.21.15.88:52526) 2023-10-18T16:07:51.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.977+0000 7f6d43fff700 1 -- 172.21.15.88:0/881420557 learned_addr learned my addr 172.21.15.88:0/881420557 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:51.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.977+0000 7f6d43fff700 1 -- 172.21.15.88:0/881420557 >> v1:172.21.15.151:6789/0 conn(0x7f6d4c14d880 legacy=0x7f6d4c0c1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:51.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.977+0000 7f6d43fff700 1 -- 172.21.15.88:0/881420557 >> v1:172.21.15.88:6789/0 conn(0x7f6d4c150340 legacy=0x7f6d4c1507d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:51.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.977+0000 7f6d43fff700 1 -- 172.21.15.88:0/881420557 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d4c0a8aa0 con 0x7f6d4c0c20a0 2023-10-18T16:07:51.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.977+0000 7f6d437fe700 1 -- 172.21.15.88:0/881420557 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2827077780 0 0) 0x7f6d4c0a8570 con 0x7f6d4c150340 2023-10-18T16:07:51.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.978+0000 7f6d43fff700 1 --2- 172.21.15.88:0/881420557 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d4c0c20a0 0x7f6d4c154280 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f6d3c006fd0 tx=0x7f6d3c0090f0).ready entity=mon.0 client_cookie=e8a069865461d37c server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:51.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.978+0000 7f6d437fe700 1 -- 172.21.15.88:0/881420557 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d3c00b600 con 0x7f6d4c0c20a0 2023-10-18T16:07:51.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.978+0000 7f6d437fe700 1 -- 172.21.15.88:0/881420557 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d3c00b760 con 0x7f6d4c0c20a0 2023-10-18T16:07:51.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.978+0000 7f6d437fe700 1 -- 172.21.15.88:0/881420557 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d3c00bae0 con 0x7f6d4c0c20a0 2023-10-18T16:07:51.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.978+0000 7f6d52fcb700 1 -- 172.21.15.88:0/881420557 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d4c0c20a0 msgr2=0x7f6d4c154280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:51.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.978+0000 7f6d52fcb700 1 --2- 172.21.15.88:0/881420557 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d4c0c20a0 0x7f6d4c154280 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f6d3c006fd0 tx=0x7f6d3c0090f0).stop 2023-10-18T16:07:51.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.978+0000 7f6d52fcb700 1 -- 172.21.15.88:0/881420557 shutdown_connections 2023-10-18T16:07:51.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.978+0000 7f6d52fcb700 1 --2- 172.21.15.88:0/881420557 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d4c0c20a0 0x7f6d4c154280 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:51.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.978+0000 7f6d52fcb700 1 -- 172.21.15.88:0/881420557 >> 172.21.15.88:0/881420557 conn(0x7f6d4c069d80 msgr2=0x7f6d4c06a190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:51.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.978+0000 7f6d52fcb700 1 -- 172.21.15.88:0/881420557 shutdown_connections 2023-10-18T16:07:51.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.978+0000 7f6d52fcb700 1 -- 172.21.15.88:0/881420557 wait complete. 2023-10-18T16:07:51.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.979+0000 7f6d52fcb700 1 Processor -- start 2023-10-18T16:07:51.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.979+0000 7f6d52fcb700 1 -- start start 2023-10-18T16:07:51.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.980+0000 7f6d52fcb700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d4c0c20a0 0x7f6d4c0abed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:51.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.980+0000 7f6d52fcb700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6d4c14d880 0x7f6d4c160bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:51.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.980+0000 7f6d52fcb700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6d4c150340 0x7f6d4c0be950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:51.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.980+0000 7f6d52fcb700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d4c0c0f20 con 0x7f6d4c0c20a0 2023-10-18T16:07:51.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.980+0000 7f6d52fcb700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d4c163eb0 con 0x7f6d4c150340 2023-10-18T16:07:51.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.980+0000 7f6d52fcb700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d4c1610a0 con 0x7f6d4c14d880 2023-10-18T16:07:51.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.980+0000 7f6d50d67700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d4c0c20a0 0x7f6d4c0abed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:51.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.980+0000 7f6d50d67700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d4c0c20a0 0x7f6d4c0abed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59652/0 (socket says 172.21.15.88:59652) 2023-10-18T16:07:51.992 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.980+0000 7f6d50d67700 1 -- 172.21.15.88:0/3634496056 learned_addr learned my addr 172.21.15.88:0/3634496056 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:51.992 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.980+0000 7f6d43fff700 1 --2- 172.21.15.88:0/3634496056 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6d4c14d880 0x7f6d4c160bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:51.992 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.980+0000 7f6d51568700 1 --2- 172.21.15.88:0/3634496056 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6d4c150340 0x7f6d4c0be950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:51.992 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.981+0000 7f6d50d67700 1 -- 172.21.15.88:0/3634496056 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6d4c14d880 msgr2=0x7f6d4c160bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:51.992 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.981+0000 7f6d50d67700 1 --2- 172.21.15.88:0/3634496056 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6d4c14d880 0x7f6d4c160bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:51.992 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.981+0000 7f6d50d67700 1 -- 172.21.15.88:0/3634496056 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6d4c150340 msgr2=0x7f6d4c0be950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:51.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.981+0000 7f6d50d67700 1 --2- 172.21.15.88:0/3634496056 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6d4c150340 0x7f6d4c0be950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:51.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.981+0000 7f6d50d67700 1 -- 172.21.15.88:0/3634496056 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d3c009370 con 0x7f6d4c0c20a0 2023-10-18T16:07:51.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.981+0000 7f6d43fff700 1 --2- 172.21.15.88:0/3634496056 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6d4c14d880 0x7f6d4c160bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-10-18T16:07:51.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.981+0000 7f6d51568700 1 --2- 172.21.15.88:0/3634496056 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6d4c150340 0x7f6d4c0be950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-10-18T16:07:51.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.981+0000 7f6d50d67700 1 --2- 172.21.15.88:0/3634496056 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d4c0c20a0 0x7f6d4c0abed0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f6d4800c840 tx=0x7f6d48001660).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:51.994 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.981+0000 7f6d40ff9700 1 -- 172.21.15.88:0/3634496056 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d48006a20 con 0x7f6d4c0c20a0 2023-10-18T16:07:51.994 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.981+0000 7f6d40ff9700 1 -- 172.21.15.88:0/3634496056 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d48006b80 con 0x7f6d4c0c20a0 2023-10-18T16:07:51.994 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.981+0000 7f6d40ff9700 1 -- 172.21.15.88:0/3634496056 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d4800a610 con 0x7f6d4c0c20a0 2023-10-18T16:07:51.994 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.981+0000 7f6d52fcb700 1 -- 172.21.15.88:0/3634496056 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d4c0ac4a0 con 0x7f6d4c0c20a0 2023-10-18T16:07:51.994 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.981+0000 7f6d52fcb700 1 -- 172.21.15.88:0/3634496056 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d4c163860 con 0x7f6d4c0c20a0 2023-10-18T16:07:51.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.982+0000 7f6d52fcb700 1 -- 172.21.15.88:0/3634496056 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d4c1636c0 con 0x7f6d4c0c20a0 2023-10-18T16:07:51.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.982+0000 7f6d40ff9700 1 -- 172.21.15.88:0/3634496056 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f6d4800aa00 con 0x7f6d4c0c20a0 2023-10-18T16:07:51.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.982+0000 7f6d40ff9700 1 --2- 172.21.15.88:0/3634496056 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f6d3805f150 0x7f6d38061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:51.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.983+0000 7f6d40ff9700 1 -- 172.21.15.88:0/3634496056 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6121+0+0 (secure 0 0 0) 0x7f6d48079f50 con 0x7f6d4c0c20a0 2023-10-18T16:07:51.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.983+0000 7f6d43fff700 1 --2- 172.21.15.88:0/3634496056 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f6d3805f150 0x7f6d38061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:51.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.983+0000 7f6d43fff700 1 --2- 172.21.15.88:0/3634496056 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f6d3805f150 0x7f6d38061610 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f6d3c006fd0 tx=0x7f6d3c017000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:51.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:51.986+0000 7f6d40ff9700 1 -- 172.21.15.88:0/3634496056 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f6d4c1636c0 con 0x7f6d4c0c20a0 2023-10-18T16:07:52.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.151+0000 7f6d52fcb700 1 -- 172.21.15.88:0/3634496056 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f6d4c1636c0 con 0x7f6d4c0c20a0 2023-10-18T16:07:52.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.152+0000 7f6d40ff9700 1 -- 172.21.15.88:0/3634496056 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6d4c1636c0 con 0x7f6d4c0c20a0 2023-10-18T16:07:52.156 INFO:teuthology.orchestra.run.smithi088.stdout:55834574853 2023-10-18T16:07:52.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.153+0000 7f6d52fcb700 1 -- 172.21.15.88:0/3634496056 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f6d3805f150 msgr2=0x7f6d38061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:52.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.153+0000 7f6d52fcb700 1 --2- 172.21.15.88:0/3634496056 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f6d3805f150 0x7f6d38061610 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f6d3c006fd0 tx=0x7f6d3c017000).stop 2023-10-18T16:07:52.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.153+0000 7f6d52fcb700 1 -- 172.21.15.88:0/3634496056 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d4c0c20a0 msgr2=0x7f6d4c0abed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:52.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.153+0000 7f6d52fcb700 1 --2- 172.21.15.88:0/3634496056 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d4c0c20a0 0x7f6d4c0abed0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f6d4800c840 tx=0x7f6d48001660).stop 2023-10-18T16:07:52.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.154+0000 7f6d52fcb700 1 -- 172.21.15.88:0/3634496056 shutdown_connections 2023-10-18T16:07:52.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.154+0000 7f6d52fcb700 1 --2- 172.21.15.88:0/3634496056 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f6d3805f150 0x7f6d38061610 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:52.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.154+0000 7f6d52fcb700 1 --2- 172.21.15.88:0/3634496056 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6d4c150340 0x7f6d4c0be950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:52.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.154+0000 7f6d52fcb700 1 --2- 172.21.15.88:0/3634496056 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d4c0c20a0 0x7f6d4c0abed0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:52.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.154+0000 7f6d52fcb700 1 --2- 172.21.15.88:0/3634496056 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6d4c14d880 0x7f6d4c160bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:52.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.154+0000 7f6d52fcb700 1 -- 172.21.15.88:0/3634496056 >> 172.21.15.88:0/3634496056 conn(0x7f6d4c069d80 msgr2=0x7f6d4c1537c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:52.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.154+0000 7f6d52fcb700 1 -- 172.21.15.88:0/3634496056 shutdown_connections 2023-10-18T16:07:52.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.154+0000 7f6d52fcb700 1 -- 172.21.15.88:0/3634496056 wait complete. 2023-10-18T16:07:52.169 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.2 2023-10-18T16:07:52.169 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-10-18T16:07:52.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.280+0000 7f1452481700 1 Processor -- start 2023-10-18T16:07:52.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.281+0000 7f1452481700 1 -- start start 2023-10-18T16:07:52.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.281+0000 7f1452481700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f144c149d50 0x7f144c067110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:52.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.281+0000 7f1452481700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f144c14a130 0x7f144c067600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:52.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.281+0000 7f1452481700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f144c151940 0x7f144c151db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:52.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.282+0000 7f1452481700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f144c0badb0 con 0x7f144c151940 2023-10-18T16:07:52.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.282+0000 7f1452481700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f144c1514b0 con 0x7f144c14a130 2023-10-18T16:07:52.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.282+0000 7f1452481700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f144c068440 con 0x7f144c149d50 2023-10-18T16:07:52.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.282+0000 7f1451c80700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f144c151940 0x7f144c151db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:52.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.282+0000 7f1451c80700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f144c151940 0x7f144c151db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59662/0 (socket says 172.21.15.88:59662) 2023-10-18T16:07:52.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.282+0000 7f1451c80700 1 -- 172.21.15.88:0/3273830373 learned_addr learned my addr 172.21.15.88:0/3273830373 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:52.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.282+0000 7f145147f700 1 --2- 172.21.15.88:0/3273830373 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f144c149d50 0x7f144c067110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:52.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.282+0000 7f1450c7e700 1 --2- 172.21.15.88:0/3273830373 >> v2:172.21.15.151:3300/0 conn(0x7f144c14a130 0x7f144c067600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:52.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.282+0000 7f1451c80700 1 -- 172.21.15.88:0/3273830373 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f144c149d50 msgr2=0x7f144c067110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:52.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.282+0000 7f1451c80700 1 --2- 172.21.15.88:0/3273830373 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f144c149d50 0x7f144c067110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:52.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.282+0000 7f1451c80700 1 -- 172.21.15.88:0/3273830373 >> v2:172.21.15.151:3300/0 conn(0x7f144c14a130 msgr2=0x7f144c067600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:52.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.282+0000 7f1451c80700 1 --2- 172.21.15.88:0/3273830373 >> v2:172.21.15.151:3300/0 conn(0x7f144c14a130 0x7f144c067600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:52.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.282+0000 7f1451c80700 1 -- 172.21.15.88:0/3273830373 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f144c0686e0 con 0x7f144c151940 2023-10-18T16:07:52.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.282+0000 7f1451c80700 1 --2- 172.21.15.88:0/3273830373 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f144c151940 0x7f144c151db0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f144c063170 tx=0x7f14440090f0).ready entity=mon.0 client_cookie=7e8fa32516032368 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:52.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.282+0000 7f1443fff700 1 -- 172.21.15.88:0/3273830373 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f144400b690 con 0x7f144c151940 2023-10-18T16:07:52.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.282+0000 7f1443fff700 1 -- 172.21.15.88:0/3273830373 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f144400b7f0 con 0x7f144c151940 2023-10-18T16:07:52.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.282+0000 7f1443fff700 1 -- 172.21.15.88:0/3273830373 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f144400bb70 con 0x7f144c151940 2023-10-18T16:07:52.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.283+0000 7f1452481700 1 -- 172.21.15.88:0/3273830373 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f144c151940 msgr2=0x7f144c151db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:52.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.283+0000 7f1452481700 1 --2- 172.21.15.88:0/3273830373 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f144c151940 0x7f144c151db0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f144c063170 tx=0x7f14440090f0).stop 2023-10-18T16:07:52.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.283+0000 7f1452481700 1 -- 172.21.15.88:0/3273830373 shutdown_connections 2023-10-18T16:07:52.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.283+0000 7f1452481700 1 --2- 172.21.15.88:0/3273830373 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f144c151940 0x7f144c151db0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:52.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.283+0000 7f1452481700 1 --2- 172.21.15.88:0/3273830373 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f144c149d50 0x7f144c067110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:52.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.283+0000 7f1452481700 1 --2- 172.21.15.88:0/3273830373 >> v2:172.21.15.151:3300/0 conn(0x7f144c14a130 0x7f144c067600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:52.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.283+0000 7f1452481700 1 -- 172.21.15.88:0/3273830373 >> 172.21.15.88:0/3273830373 conn(0x7f144c062650 msgr2=0x7f144c062a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:52.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.283+0000 7f1452481700 1 -- 172.21.15.88:0/3273830373 shutdown_connections 2023-10-18T16:07:52.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.283+0000 7f1452481700 1 -- 172.21.15.88:0/3273830373 wait complete. 2023-10-18T16:07:52.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.284+0000 7f1452481700 1 Processor -- start 2023-10-18T16:07:52.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.284+0000 7f1452481700 1 -- start start 2023-10-18T16:07:52.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.285+0000 7f1452481700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f144c149d50 0x7f144c150ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:52.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.285+0000 7f1452481700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f144c14a130 0x7f144c163e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:52.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.285+0000 7f1452481700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f144c151940 0x7f144c064a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:52.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.285+0000 7f1452481700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f144c0bbd90 con 0x7f144c151940 2023-10-18T16:07:52.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.285+0000 7f1452481700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f144c065810 con 0x7f144c14a130 2023-10-18T16:07:52.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.285+0000 7f1452481700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f144c0be7a0 con 0x7f144c149d50 2023-10-18T16:07:52.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.285+0000 7f1451c80700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f144c151940 0x7f144c064a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:52.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.285+0000 7f1451c80700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f144c151940 0x7f144c064a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59668/0 (socket says 172.21.15.88:59668) 2023-10-18T16:07:52.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.285+0000 7f1451c80700 1 -- 172.21.15.88:0/1102332473 learned_addr learned my addr 172.21.15.88:0/1102332473 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:52.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.285+0000 7f145147f700 1 --2- 172.21.15.88:0/1102332473 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f144c149d50 0x7f144c150ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:52.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.285+0000 7f1450c7e700 1 --2- 172.21.15.88:0/1102332473 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f144c14a130 0x7f144c163e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:52.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.285+0000 7f1451c80700 1 -- 172.21.15.88:0/1102332473 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f144c149d50 msgr2=0x7f144c150ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:52.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.285+0000 7f1451c80700 1 --2- 172.21.15.88:0/1102332473 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f144c149d50 0x7f144c150ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:52.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.285+0000 7f1451c80700 1 -- 172.21.15.88:0/1102332473 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f144c14a130 msgr2=0x7f144c163e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:52.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.285+0000 7f1451c80700 1 --2- 172.21.15.88:0/1102332473 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f144c14a130 0x7f144c163e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:52.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.285+0000 7f1451c80700 1 -- 172.21.15.88:0/1102332473 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1444009370 con 0x7f144c151940 2023-10-18T16:07:52.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.285+0000 7f1451c80700 1 --2- 172.21.15.88:0/1102332473 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f144c151940 0x7f144c064a60 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f144c0695b0 tx=0x7f1444017000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:52.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.285+0000 7f14417fa700 1 -- 172.21.15.88:0/1102332473 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f144400b690 con 0x7f144c151940 2023-10-18T16:07:52.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.285+0000 7f14417fa700 1 -- 172.21.15.88:0/1102332473 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f144401c420 con 0x7f144c151940 2023-10-18T16:07:52.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.285+0000 7f1452481700 1 -- 172.21.15.88:0/1102332473 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f144c163970 con 0x7f144c151940 2023-10-18T16:07:52.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.285+0000 7f14417fa700 1 -- 172.21.15.88:0/1102332473 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f144400bb70 con 0x7f144c151940 2023-10-18T16:07:52.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.285+0000 7f1452481700 1 -- 172.21.15.88:0/1102332473 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f144c160db0 con 0x7f144c151940 2023-10-18T16:07:52.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.286+0000 7f1452481700 1 -- 172.21.15.88:0/1102332473 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f144c069fd0 con 0x7f144c151940 2023-10-18T16:07:52.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.287+0000 7f14417fa700 1 -- 172.21.15.88:0/1102332473 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f144401c9f0 con 0x7f144c151940 2023-10-18T16:07:52.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.287+0000 7f14417fa700 1 --2- 172.21.15.88:0/1102332473 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f143005f150 0x7f1430061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:52.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.287+0000 7f14417fa700 1 -- 172.21.15.88:0/1102332473 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6121+0+0 (secure 0 0 0) 0x7f144407b6b0 con 0x7f144c151940 2023-10-18T16:07:52.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.287+0000 7f145147f700 1 --2- 172.21.15.88:0/1102332473 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f143005f150 0x7f1430061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:52.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.288+0000 7f145147f700 1 --2- 172.21.15.88:0/1102332473 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f143005f150 0x7f1430061610 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f1448009fd0 tx=0x7f1448009880).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:52.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.291+0000 7f14417fa700 1 -- 172.21.15.88:0/1102332473 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f144404c980 con 0x7f144c151940 2023-10-18T16:07:52.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.455+0000 7f1452481700 1 -- 172.21.15.88:0/1102332473 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f144c0689c0 con 0x7f144c151940 2023-10-18T16:07:52.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.455+0000 7f14417fa700 1 -- 172.21.15.88:0/1102332473 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1444022070 con 0x7f144c151940 2023-10-18T16:07:52.459 INFO:teuthology.orchestra.run.smithi088.stdout:55834574854 2023-10-18T16:07:52.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.457+0000 7f1452481700 1 -- 172.21.15.88:0/1102332473 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f143005f150 msgr2=0x7f1430061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:52.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.457+0000 7f1452481700 1 --2- 172.21.15.88:0/1102332473 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f143005f150 0x7f1430061610 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f1448009fd0 tx=0x7f1448009880).stop 2023-10-18T16:07:52.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.457+0000 7f1452481700 1 -- 172.21.15.88:0/1102332473 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f144c151940 msgr2=0x7f144c064a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:52.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.457+0000 7f1452481700 1 --2- 172.21.15.88:0/1102332473 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f144c151940 0x7f144c064a60 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f144c0695b0 tx=0x7f1444017000).stop 2023-10-18T16:07:52.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.457+0000 7f1452481700 1 -- 172.21.15.88:0/1102332473 shutdown_connections 2023-10-18T16:07:52.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.457+0000 7f1452481700 1 --2- 172.21.15.88:0/1102332473 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f143005f150 0x7f1430061610 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:52.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.457+0000 7f1452481700 1 --2- 172.21.15.88:0/1102332473 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f144c151940 0x7f144c064a60 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:52.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.457+0000 7f1452481700 1 --2- 172.21.15.88:0/1102332473 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f144c149d50 0x7f144c150ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:52.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.457+0000 7f1452481700 1 --2- 172.21.15.88:0/1102332473 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f144c14a130 0x7f144c163e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:52.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.457+0000 7f1452481700 1 -- 172.21.15.88:0/1102332473 >> 172.21.15.88:0/1102332473 conn(0x7f144c062650 msgr2=0x7f144c151d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:52.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.457+0000 7f1452481700 1 -- 172.21.15.88:0/1102332473 shutdown_connections 2023-10-18T16:07:52.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.457+0000 7f1452481700 1 -- 172.21.15.88:0/1102332473 wait complete. 2023-10-18T16:07:52.473 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.3 2023-10-18T16:07:52.474 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-10-18T16:07:52.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.585+0000 7fa65e5ab700 1 Processor -- start 2023-10-18T16:07:52.589 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.585+0000 7fa65e5ab700 1 -- start start 2023-10-18T16:07:52.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.585+0000 7fa65e5ab700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa65814d880 0x7fa658154280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:52.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.586+0000 7fa65e5ab700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa6580c1990 0x7fa6580c1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:52.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.586+0000 7fa65e5ab700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6580a8570 con 0x7fa658064490 2023-10-18T16:07:52.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.586+0000 7fa65e5ab700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa658154770 con 0x7fa65814d880 2023-10-18T16:07:52.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.586+0000 7fa65e5ab700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6580a87c0 con 0x7fa6580c1990 2023-10-18T16:07:52.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.586+0000 7fa657fff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa65814d880 0x7fa658154280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:52.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.586+0000 7fa657fff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa65814d880 0x7fa658154280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59676/0 (socket says 172.21.15.88:59676) 2023-10-18T16:07:52.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.586+0000 7fa657fff700 1 -- 172.21.15.88:0/2426817091 learned_addr learned my addr 172.21.15.88:0/2426817091 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:52.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.586+0000 7fa657fff700 1 -- 172.21.15.88:0/2426817091 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa6580c1990 msgr2=0x7fa6580c1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:52.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.586+0000 7fa657fff700 1 --2- 172.21.15.88:0/2426817091 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa6580c1990 0x7fa6580c1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:07:52.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.586+0000 7fa657fff700 1 -- 172.21.15.88:0/2426817091 >> v1:172.21.15.151:6789/0 conn(0x7fa658064490 legacy=0x7fa658064920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:52.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.586+0000 7fa657fff700 1 -- 172.21.15.88:0/2426817091 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6580a8aa0 con 0x7fa65814d880 2023-10-18T16:07:52.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.586+0000 7fa657fff700 1 --2- 172.21.15.88:0/2426817091 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa65814d880 0x7fa658154280 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fa64c006fd0 tx=0x7fa64c0090f0).ready entity=mon.0 client_cookie=5746f1ef08509c35 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:52.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.586+0000 7fa656ffd700 1 -- 172.21.15.88:0/2426817091 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa64c00b600 con 0x7fa65814d880 2023-10-18T16:07:52.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.586+0000 7fa656ffd700 1 -- 172.21.15.88:0/2426817091 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa64c00b760 con 0x7fa65814d880 2023-10-18T16:07:52.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.586+0000 7fa656ffd700 1 -- 172.21.15.88:0/2426817091 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa64c00bae0 con 0x7fa65814d880 2023-10-18T16:07:52.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.587+0000 7fa65e5ab700 1 -- 172.21.15.88:0/2426817091 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa65814d880 msgr2=0x7fa658154280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:52.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.587+0000 7fa65e5ab700 1 --2- 172.21.15.88:0/2426817091 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa65814d880 0x7fa658154280 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fa64c006fd0 tx=0x7fa64c0090f0).stop 2023-10-18T16:07:52.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.587+0000 7fa65e5ab700 1 -- 172.21.15.88:0/2426817091 shutdown_connections 2023-10-18T16:07:52.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.587+0000 7fa65e5ab700 1 --2- 172.21.15.88:0/2426817091 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa65814d880 0x7fa658154280 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:52.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.587+0000 7fa65e5ab700 1 --2- 172.21.15.88:0/2426817091 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa6580c1990 0x7fa6580c1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:07:52.599 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.587+0000 7fa65e5ab700 1 -- 172.21.15.88:0/2426817091 >> 172.21.15.88:0/2426817091 conn(0x7fa658069d80 msgr2=0x7fa65806a190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:52.599 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.587+0000 7fa65e5ab700 1 -- 172.21.15.88:0/2426817091 shutdown_connections 2023-10-18T16:07:52.599 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.587+0000 7fa65e5ab700 1 -- 172.21.15.88:0/2426817091 wait complete. 2023-10-18T16:07:52.599 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.588+0000 7fa65e5ab700 1 Processor -- start 2023-10-18T16:07:52.599 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.588+0000 7fa65e5ab700 1 -- start start 2023-10-18T16:07:52.599 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.589+0000 7fa65e5ab700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa658064490 0x7fa65814f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:52.600 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.589+0000 7fa65e5ab700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6580c1990 0x7fa65814c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:52.600 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.589+0000 7fa65e5ab700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa65814d880 0x7fa658152680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:52.600 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.589+0000 7fa65e5ab700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6580c1180 con 0x7fa6580c1990 2023-10-18T16:07:52.600 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.589+0000 7fa65e5ab700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa658163820 con 0x7fa65814d880 2023-10-18T16:07:52.600 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.589+0000 7fa65e5ab700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6581514b0 con 0x7fa658064490 2023-10-18T16:07:52.601 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.589+0000 7fa6577fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6580c1990 0x7fa65814c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:52.601 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.589+0000 7fa6577fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6580c1990 0x7fa65814c020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59682/0 (socket says 172.21.15.88:59682) 2023-10-18T16:07:52.601 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.589+0000 7fa6577fe700 1 -- 172.21.15.88:0/172020861 learned_addr learned my addr 172.21.15.88:0/172020861 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:52.601 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.589+0000 7fa657fff700 1 --2- 172.21.15.88:0/172020861 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa658064490 0x7fa65814f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:52.601 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.589+0000 7fa65cb48700 1 --2- 172.21.15.88:0/172020861 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa65814d880 0x7fa658152680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:52.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.589+0000 7fa6577fe700 1 -- 172.21.15.88:0/172020861 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa658064490 msgr2=0x7fa65814f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:52.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.589+0000 7fa6577fe700 1 --2- 172.21.15.88:0/172020861 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa658064490 0x7fa65814f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:52.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.589+0000 7fa6577fe700 1 -- 172.21.15.88:0/172020861 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa65814d880 msgr2=0x7fa658152680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:52.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.589+0000 7fa6577fe700 1 --2- 172.21.15.88:0/172020861 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa65814d880 0x7fa658152680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:52.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.589+0000 7fa6577fe700 1 -- 172.21.15.88:0/172020861 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa64c009370 con 0x7fa6580c1990 2023-10-18T16:07:52.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.589+0000 7fa6577fe700 1 --2- 172.21.15.88:0/172020861 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6580c1990 0x7fa65814c020 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fa64800b560 tx=0x7fa6480090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:52.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.589+0000 7fa63ffff700 1 -- 172.21.15.88:0/172020861 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa648002320 con 0x7fa6580c1990 2023-10-18T16:07:52.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.589+0000 7fa63ffff700 1 -- 172.21.15.88:0/172020861 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6480024a0 con 0x7fa6580c1990 2023-10-18T16:07:52.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.589+0000 7fa65e5ab700 1 -- 172.21.15.88:0/172020861 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6580c0ea0 con 0x7fa6580c1990 2023-10-18T16:07:52.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.589+0000 7fa63ffff700 1 -- 172.21.15.88:0/172020861 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa648002860 con 0x7fa6580c1990 2023-10-18T16:07:52.604 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.590+0000 7fa65e5ab700 1 -- 172.21.15.88:0/172020861 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6580bd220 con 0x7fa6580c1990 2023-10-18T16:07:52.604 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.590+0000 7fa65e5ab700 1 -- 172.21.15.88:0/172020861 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa65806c4e0 con 0x7fa6580c1990 2023-10-18T16:07:52.604 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.591+0000 7fa63ffff700 1 -- 172.21.15.88:0/172020861 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7fa648002c10 con 0x7fa6580c1990 2023-10-18T16:07:52.604 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.591+0000 7fa63ffff700 1 --2- 172.21.15.88:0/172020861 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fa64405f150 0x7fa644061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:52.604 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.591+0000 7fa63ffff700 1 -- 172.21.15.88:0/172020861 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6121+0+0 (secure 0 0 0) 0x7fa64807a020 con 0x7fa6580c1990 2023-10-18T16:07:52.605 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.591+0000 7fa657fff700 1 --2- 172.21.15.88:0/172020861 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fa64405f150 0x7fa644061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:52.605 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.592+0000 7fa657fff700 1 --2- 172.21.15.88:0/172020861 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fa64405f150 0x7fa644061610 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fa64c009130 tx=0x7fa64c017000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:52.605 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.595+0000 7fa63ffff700 1 -- 172.21.15.88:0/172020861 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7fa64804b2f0 con 0x7fa6580c1990 2023-10-18T16:07:52.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.763+0000 7fa65e5ab700 1 -- 172.21.15.88:0/172020861 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fa658163960 con 0x7fa6580c1990 2023-10-18T16:07:52.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.764+0000 7fa63ffff700 1 -- 172.21.15.88:0/172020861 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa64804e910 con 0x7fa6580c1990 2023-10-18T16:07:52.767 INFO:teuthology.orchestra.run.smithi088.stdout:55834574854 2023-10-18T16:07:52.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.765+0000 7fa65e5ab700 1 -- 172.21.15.88:0/172020861 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fa64405f150 msgr2=0x7fa644061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:52.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.765+0000 7fa65e5ab700 1 --2- 172.21.15.88:0/172020861 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fa64405f150 0x7fa644061610 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fa64c009130 tx=0x7fa64c017000).stop 2023-10-18T16:07:52.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.765+0000 7fa65e5ab700 1 -- 172.21.15.88:0/172020861 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6580c1990 msgr2=0x7fa65814c020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:52.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.765+0000 7fa65e5ab700 1 --2- 172.21.15.88:0/172020861 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6580c1990 0x7fa65814c020 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fa64800b560 tx=0x7fa6480090f0).stop 2023-10-18T16:07:52.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.766+0000 7fa65e5ab700 1 -- 172.21.15.88:0/172020861 shutdown_connections 2023-10-18T16:07:52.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.766+0000 7fa65e5ab700 1 --2- 172.21.15.88:0/172020861 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fa64405f150 0x7fa644061610 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:52.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.766+0000 7fa65e5ab700 1 --2- 172.21.15.88:0/172020861 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa65814d880 0x7fa658152680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:52.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.766+0000 7fa65e5ab700 1 --2- 172.21.15.88:0/172020861 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa658064490 0x7fa65814f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:52.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.766+0000 7fa65e5ab700 1 --2- 172.21.15.88:0/172020861 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6580c1990 0x7fa65814c020 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:52.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.766+0000 7fa65e5ab700 1 -- 172.21.15.88:0/172020861 >> 172.21.15.88:0/172020861 conn(0x7fa658069d80 msgr2=0x7fa6580c5f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:52.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.766+0000 7fa65e5ab700 1 -- 172.21.15.88:0/172020861 shutdown_connections 2023-10-18T16:07:52.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.766+0000 7fa65e5ab700 1 -- 172.21.15.88:0/172020861 wait complete. 2023-10-18T16:07:52.781 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.4 2023-10-18T16:07:52.781 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-10-18T16:07:52.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.891+0000 7f0290aef700 1 Processor -- start 2023-10-18T16:07:52.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.891+0000 7f0290aef700 1 -- start start 2023-10-18T16:07:52.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.892+0000 7f0290aef700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f028c06a360 0x7f028c0b9370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:52.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.892+0000 7f0290aef700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f028c0b9860 0x7f028c157810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:52.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.892+0000 7f0290aef700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f028c069b40 con 0x7f028c0bf270 2023-10-18T16:07:52.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.892+0000 7f0290aef700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f028c14cea0 con 0x7f028c06a360 2023-10-18T16:07:52.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.892+0000 7f0290aef700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f028c14d140 con 0x7f028c0b9860 2023-10-18T16:07:52.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.892+0000 7f028bfff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f028c0b9860 0x7f028c157810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:52.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.892+0000 7f028affd700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f028c06a360 0x7f028c0b9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:52.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.892+0000 7f028affd700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f028c06a360 0x7f028c0b9370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.88:60240/0 (socket says 172.21.15.88:60240) 2023-10-18T16:07:52.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.892+0000 7f028bfff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f028c0b9860 0x7f028c157810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.88:54016/0 (socket says 172.21.15.88:54016) 2023-10-18T16:07:52.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.892+0000 7f028affd700 1 -- 172.21.15.88:0/3542476283 learned_addr learned my addr 172.21.15.88:0/3542476283 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:52.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.893+0000 7f028affd700 1 -- 172.21.15.88:0/3542476283 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f028c0b9860 msgr2=0x7f028c157810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:52.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.893+0000 7f028affd700 1 --2- 172.21.15.88:0/3542476283 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f028c0b9860 0x7f028c157810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:52.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.893+0000 7f028affd700 1 -- 172.21.15.88:0/3542476283 >> v1:172.21.15.151:6789/0 conn(0x7f028c0bf270 legacy=0x7f028c14d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:52.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.893+0000 7f028affd700 1 -- 172.21.15.88:0/3542476283 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f028c0be5d0 con 0x7f028c06a360 2023-10-18T16:07:52.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.893+0000 7f028affd700 1 --2- 172.21.15.88:0/3542476283 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f028c06a360 0x7f028c0b9370 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f0280006fd0 tx=0x7f02800090f0).ready entity=mon.1 client_cookie=fd1e0df10bd67215 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:52.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.893+0000 7f028a7fc700 1 -- 172.21.15.88:0/3542476283 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f028000b580 con 0x7f028c06a360 2023-10-18T16:07:52.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.893+0000 7f028a7fc700 1 -- 172.21.15.88:0/3542476283 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f028000b6e0 con 0x7f028c06a360 2023-10-18T16:07:52.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.893+0000 7f028a7fc700 1 -- 172.21.15.88:0/3542476283 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f028000ba60 con 0x7f028c06a360 2023-10-18T16:07:52.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.893+0000 7f0290aef700 1 -- 172.21.15.88:0/3542476283 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f028c06a360 msgr2=0x7f028c0b9370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:52.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.893+0000 7f0290aef700 1 --2- 172.21.15.88:0/3542476283 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f028c06a360 0x7f028c0b9370 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f0280006fd0 tx=0x7f02800090f0).stop 2023-10-18T16:07:52.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.894+0000 7f0290aef700 1 -- 172.21.15.88:0/3542476283 shutdown_connections 2023-10-18T16:07:52.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.894+0000 7f0290aef700 1 --2- 172.21.15.88:0/3542476283 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f028c0b9860 0x7f028c157810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:52.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.894+0000 7f0290aef700 1 --2- 172.21.15.88:0/3542476283 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f028c06a360 0x7f028c0b9370 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:52.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.894+0000 7f0290aef700 1 -- 172.21.15.88:0/3542476283 >> 172.21.15.88:0/3542476283 conn(0x7f028c065e10 msgr2=0x7f028c0c2cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:52.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.894+0000 7f0290aef700 1 -- 172.21.15.88:0/3542476283 shutdown_connections 2023-10-18T16:07:52.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.894+0000 7f0290aef700 1 -- 172.21.15.88:0/3542476283 wait complete. 2023-10-18T16:07:52.905 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.895+0000 7f0290aef700 1 Processor -- start 2023-10-18T16:07:52.906 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.895+0000 7f0290aef700 1 -- start start 2023-10-18T16:07:52.906 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.895+0000 7f0290aef700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f028c06a360 0x7f028c154a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:52.906 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.895+0000 7f0290aef700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f028c0b9860 0x7f028c150c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:52.906 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.896+0000 7f0290aef700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f028c0bf270 0x7f028c160df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:52.906 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.896+0000 7f0290aef700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f028c149060 con 0x7f028c0bf270 2023-10-18T16:07:52.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.896+0000 7f0290aef700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f028c150780 con 0x7f028c06a360 2023-10-18T16:07:52.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.896+0000 7f0290aef700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f028c14ea90 con 0x7f028c0b9860 2023-10-18T16:07:52.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.896+0000 7f028bfff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f028c0bf270 0x7f028c160df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:52.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.896+0000 7f028bfff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f028c0bf270 0x7f028c160df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59690/0 (socket says 172.21.15.88:59690) 2023-10-18T16:07:52.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.896+0000 7f028bfff700 1 -- 172.21.15.88:0/2447349967 learned_addr learned my addr 172.21.15.88:0/2447349967 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:52.908 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.896+0000 7f028affd700 1 --2- 172.21.15.88:0/2447349967 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f028c0b9860 0x7f028c150c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:52.908 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.896+0000 7f028b7fe700 1 --2- 172.21.15.88:0/2447349967 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f028c06a360 0x7f028c154a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:52.908 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.896+0000 7f028bfff700 1 -- 172.21.15.88:0/2447349967 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f028c0b9860 msgr2=0x7f028c150c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:52.908 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.896+0000 7f028bfff700 1 --2- 172.21.15.88:0/2447349967 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f028c0b9860 0x7f028c150c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:52.908 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.896+0000 7f028bfff700 1 -- 172.21.15.88:0/2447349967 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f028c06a360 msgr2=0x7f028c154a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:52.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.896+0000 7f028bfff700 1 --2- 172.21.15.88:0/2447349967 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f028c06a360 0x7f028c154a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:52.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.896+0000 7f028bfff700 1 -- 172.21.15.88:0/2447349967 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0280009370 con 0x7f028c0bf270 2023-10-18T16:07:52.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.896+0000 7f028bfff700 1 --2- 172.21.15.88:0/2447349967 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f028c0bf270 0x7f028c160df0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f028400e930 tx=0x7f0284009060).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:52.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.896+0000 7f02737fe700 1 -- 172.21.15.88:0/2447349967 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02840093b0 con 0x7f028c0bf270 2023-10-18T16:07:52.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.896+0000 7f02737fe700 1 -- 172.21.15.88:0/2447349967 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0284012600 con 0x7f028c0bf270 2023-10-18T16:07:52.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.896+0000 7f0290aef700 1 -- 172.21.15.88:0/2447349967 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f028c149b30 con 0x7f028c0bf270 2023-10-18T16:07:52.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.896+0000 7f02737fe700 1 -- 172.21.15.88:0/2447349967 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0284012980 con 0x7f028c0bf270 2023-10-18T16:07:52.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.896+0000 7f0290aef700 1 -- 172.21.15.88:0/2447349967 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f028c1492b0 con 0x7f028c0bf270 2023-10-18T16:07:52.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.897+0000 7f0290aef700 1 -- 172.21.15.88:0/2447349967 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f028c14dc90 con 0x7f028c0bf270 2023-10-18T16:07:52.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.898+0000 7f02737fe700 1 -- 172.21.15.88:0/2447349967 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f028401b400 con 0x7f028c0bf270 2023-10-18T16:07:52.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.898+0000 7f02737fe700 1 --2- 172.21.15.88:0/2447349967 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f027805f180 0x7f0278061640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:52.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.898+0000 7f02737fe700 1 -- 172.21.15.88:0/2447349967 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6121+0+0 (secure 0 0 0) 0x7f028407a3e0 con 0x7f028c0bf270 2023-10-18T16:07:52.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.898+0000 7f028b7fe700 1 --2- 172.21.15.88:0/2447349967 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f027805f180 0x7f0278061640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:52.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.898+0000 7f028b7fe700 1 --2- 172.21.15.88:0/2447349967 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f027805f180 0x7f0278061640 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f027c004290 tx=0x7f027c0014c0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:52.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:52.902+0000 7f02737fe700 1 -- 172.21.15.88:0/2447349967 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f028404b650 con 0x7f028c0bf270 2023-10-18T16:07:53.067 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.063+0000 7f0290aef700 1 -- 172.21.15.88:0/2447349967 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f028c0c33e0 con 0x7f028c0bf270 2023-10-18T16:07:53.067 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.063+0000 7f02737fe700 1 -- 172.21.15.88:0/2447349967 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f028404ec70 con 0x7f028c0bf270 2023-10-18T16:07:53.067 INFO:teuthology.orchestra.run.smithi088.stdout:55834574853 2023-10-18T16:07:53.069 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.065+0000 7f0290aef700 1 -- 172.21.15.88:0/2447349967 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f027805f180 msgr2=0x7f0278061640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:53.069 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.065+0000 7f0290aef700 1 --2- 172.21.15.88:0/2447349967 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f027805f180 0x7f0278061640 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f027c004290 tx=0x7f027c0014c0).stop 2023-10-18T16:07:53.070 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.065+0000 7f0290aef700 1 -- 172.21.15.88:0/2447349967 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f028c0bf270 msgr2=0x7f028c160df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:53.070 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.065+0000 7f0290aef700 1 --2- 172.21.15.88:0/2447349967 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f028c0bf270 0x7f028c160df0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f028400e930 tx=0x7f0284009060).stop 2023-10-18T16:07:53.070 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.065+0000 7f0290aef700 1 -- 172.21.15.88:0/2447349967 shutdown_connections 2023-10-18T16:07:53.070 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.065+0000 7f0290aef700 1 --2- 172.21.15.88:0/2447349967 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f027805f180 0x7f0278061640 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:53.070 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.065+0000 7f0290aef700 1 --2- 172.21.15.88:0/2447349967 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f028c0bf270 0x7f028c160df0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:53.071 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.065+0000 7f0290aef700 1 --2- 172.21.15.88:0/2447349967 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f028c06a360 0x7f028c154a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:53.071 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.065+0000 7f0290aef700 1 --2- 172.21.15.88:0/2447349967 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f028c0b9860 0x7f028c150c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:53.071 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.065+0000 7f0290aef700 1 -- 172.21.15.88:0/2447349967 >> 172.21.15.88:0/2447349967 conn(0x7f028c065e10 msgr2=0x7f028c0c2cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:53.071 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.065+0000 7f0290aef700 1 -- 172.21.15.88:0/2447349967 shutdown_connections 2023-10-18T16:07:53.071 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.065+0000 7f0290aef700 1 -- 172.21.15.88:0/2447349967 wait complete. 2023-10-18T16:07:53.081 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.5 2023-10-18T16:07:53.081 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-10-18T16:07:53.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.192+0000 7f11bd41d700 1 Processor -- start 2023-10-18T16:07:53.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.192+0000 7f11bd41d700 1 -- start start 2023-10-18T16:07:53.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.193+0000 7f11bd41d700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f11b80bbef0 0x7f11b80bc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:53.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.193+0000 7f11bd41d700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f11b80c0cf0 0x7f11b814ec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:53.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.193+0000 7f11bd41d700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f11b80be0f0 0x7f11b80be5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:53.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.193+0000 7f11bd41d700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11b80bdc80 con 0x7f11b80c0cf0 2023-10-18T16:07:53.204 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.193+0000 7f11bd41d700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11b80bea90 con 0x7f11b80be0f0 2023-10-18T16:07:53.204 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.193+0000 7f11bd41d700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11b8066e40 con 0x7f11b80bbef0 2023-10-18T16:07:53.204 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.193+0000 7f11b67fc700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f11b80c0cf0 0x7f11b814ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:53.204 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.193+0000 7f11b67fc700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f11b80c0cf0 0x7f11b814ec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59700/0 (socket says 172.21.15.88:59700) 2023-10-18T16:07:53.204 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.193+0000 7f11b67fc700 1 -- 172.21.15.88:0/203810255 learned_addr learned my addr 172.21.15.88:0/203810255 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:53.205 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.193+0000 7f11b6ffd700 1 --2- 172.21.15.88:0/203810255 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f11b80bbef0 0x7f11b80bc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:53.205 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.193+0000 7f11b77fe700 1 --2- 172.21.15.88:0/203810255 >> v2:172.21.15.151:3300/0 conn(0x7f11b80be0f0 0x7f11b80be5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:53.205 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.193+0000 7f11b67fc700 1 -- 172.21.15.88:0/203810255 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f11b80bbef0 msgr2=0x7f11b80bc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:53.205 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.193+0000 7f11b67fc700 1 --2- 172.21.15.88:0/203810255 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f11b80bbef0 0x7f11b80bc2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:53.205 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.193+0000 7f11b67fc700 1 -- 172.21.15.88:0/203810255 >> v2:172.21.15.151:3300/0 conn(0x7f11b80be0f0 msgr2=0x7f11b80be5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:53.205 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.193+0000 7f11b67fc700 1 --2- 172.21.15.88:0/203810255 >> v2:172.21.15.151:3300/0 conn(0x7f11b80be0f0 0x7f11b80be5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:53.206 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.193+0000 7f11b67fc700 1 -- 172.21.15.88:0/203810255 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11b8067100 con 0x7f11b80c0cf0 2023-10-18T16:07:53.206 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.194+0000 7f11b67fc700 1 --2- 172.21.15.88:0/203810255 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11b80c0cf0 0x7f11b814ec70 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f11b80c27c0 tx=0x7f11ac0090f0).ready entity=mon.0 client_cookie=b1f47b0c0ba57932 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:53.206 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.194+0000 7f11b5ffb700 1 -- 172.21.15.88:0/203810255 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11ac00f510 con 0x7f11b80c0cf0 2023-10-18T16:07:53.206 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.194+0000 7f11b5ffb700 1 -- 172.21.15.88:0/203810255 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11ac00f670 con 0x7f11b80c0cf0 2023-10-18T16:07:53.206 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.194+0000 7f11b5ffb700 1 -- 172.21.15.88:0/203810255 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11ac00f9f0 con 0x7f11b80c0cf0 2023-10-18T16:07:53.207 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.194+0000 7f11bd41d700 1 -- 172.21.15.88:0/203810255 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11b80c0cf0 msgr2=0x7f11b814ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:53.207 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.194+0000 7f11bd41d700 1 --2- 172.21.15.88:0/203810255 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11b80c0cf0 0x7f11b814ec70 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f11b80c27c0 tx=0x7f11ac0090f0).stop 2023-10-18T16:07:53.207 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.194+0000 7f11bd41d700 1 -- 172.21.15.88:0/203810255 shutdown_connections 2023-10-18T16:07:53.207 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.194+0000 7f11bd41d700 1 --2- 172.21.15.88:0/203810255 >> v2:172.21.15.151:3300/0 conn(0x7f11b80be0f0 0x7f11b80be5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:53.207 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.194+0000 7f11bd41d700 1 --2- 172.21.15.88:0/203810255 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f11b80bbef0 0x7f11b80bc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:53.208 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.194+0000 7f11bd41d700 1 --2- 172.21.15.88:0/203810255 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11b80c0cf0 0x7f11b814ec70 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:53.208 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.194+0000 7f11bd41d700 1 -- 172.21.15.88:0/203810255 >> 172.21.15.88:0/203810255 conn(0x7f11b80c48f0 msgr2=0x7f11b80c4d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:53.208 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.194+0000 7f11bd41d700 1 -- 172.21.15.88:0/203810255 shutdown_connections 2023-10-18T16:07:53.208 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.194+0000 7f11bd41d700 1 -- 172.21.15.88:0/203810255 wait complete. 2023-10-18T16:07:53.208 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.195+0000 7f11bd41d700 1 Processor -- start 2023-10-18T16:07:53.209 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.196+0000 7f11bd41d700 1 -- start start 2023-10-18T16:07:53.209 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.196+0000 7f11bd41d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f11b80bbef0 0x7f11b80b8220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:53.209 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.196+0000 7f11bd41d700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f11b80be0f0 0x7f11b81520f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:53.209 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.196+0000 7f11bd41d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11b80c0cf0 0x7f11b8161d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:53.209 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.196+0000 7f11bd41d700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11b80b9820 con 0x7f11b80c0cf0 2023-10-18T16:07:53.209 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.196+0000 7f11bd41d700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11b814bb10 con 0x7f11b80bbef0 2023-10-18T16:07:53.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.196+0000 7f11bd41d700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11b814af40 con 0x7f11b80be0f0 2023-10-18T16:07:53.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.196+0000 7f11b77fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11b80c0cf0 0x7f11b8161d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:53.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.196+0000 7f11b77fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11b80c0cf0 0x7f11b8161d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59714/0 (socket says 172.21.15.88:59714) 2023-10-18T16:07:53.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.196+0000 7f11b77fe700 1 -- 172.21.15.88:0/1899913659 learned_addr learned my addr 172.21.15.88:0/1899913659 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:53.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.196+0000 7f11b6ffd700 1 --2- 172.21.15.88:0/1899913659 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f11b80bbef0 0x7f11b80b8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:53.211 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.196+0000 7f11b67fc700 1 --2- 172.21.15.88:0/1899913659 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f11b80be0f0 0x7f11b81520f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:53.211 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.197+0000 7f11b77fe700 1 -- 172.21.15.88:0/1899913659 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f11b80be0f0 msgr2=0x7f11b81520f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:53.211 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.197+0000 7f11b77fe700 1 --2- 172.21.15.88:0/1899913659 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f11b80be0f0 0x7f11b81520f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:53.211 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.197+0000 7f11b77fe700 1 -- 172.21.15.88:0/1899913659 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f11b80bbef0 msgr2=0x7f11b80b8220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:53.211 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.197+0000 7f11b77fe700 1 --2- 172.21.15.88:0/1899913659 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f11b80bbef0 0x7f11b80b8220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:53.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.197+0000 7f11b77fe700 1 -- 172.21.15.88:0/1899913659 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11ac009370 con 0x7f11b80c0cf0 2023-10-18T16:07:53.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.197+0000 7f11b6ffd700 1 --2- 172.21.15.88:0/1899913659 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f11b80bbef0 0x7f11b80b8220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:07:53.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.197+0000 7f11b67fc700 1 --2- 172.21.15.88:0/1899913659 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f11b80be0f0 0x7f11b81520f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:07:53.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.197+0000 7f11b77fe700 1 --2- 172.21.15.88:0/1899913659 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11b80c0cf0 0x7f11b8161d20 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f11b000e8d0 tx=0x7f11b0009060).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:53.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.197+0000 7f119effd700 1 -- 172.21.15.88:0/1899913659 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11b000ee70 con 0x7f11b80c0cf0 2023-10-18T16:07:53.213 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.197+0000 7f119effd700 1 -- 172.21.15.88:0/1899913659 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11b0009530 con 0x7f11b80c0cf0 2023-10-18T16:07:53.213 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.197+0000 7f11bd41d700 1 -- 172.21.15.88:0/1899913659 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11b8151650 con 0x7f11b80c0cf0 2023-10-18T16:07:53.213 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.197+0000 7f119effd700 1 -- 172.21.15.88:0/1899913659 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11b000ca60 con 0x7f11b80c0cf0 2023-10-18T16:07:53.213 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.197+0000 7f11bd41d700 1 -- 172.21.15.88:0/1899913659 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11b814b4c0 con 0x7f11b80c0cf0 2023-10-18T16:07:53.213 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.198+0000 7f11bd41d700 1 -- 172.21.15.88:0/1899913659 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11b814bca0 con 0x7f11b80c0cf0 2023-10-18T16:07:53.214 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.198+0000 7f119effd700 1 -- 172.21.15.88:0/1899913659 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f11b001b400 con 0x7f11b80c0cf0 2023-10-18T16:07:53.214 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.198+0000 7f119effd700 1 --2- 172.21.15.88:0/1899913659 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f11a405f180 0x7f11a4061640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:53.214 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.198+0000 7f119effd700 1 -- 172.21.15.88:0/1899913659 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6121+0+0 (secure 0 0 0) 0x7f11b007a6d0 con 0x7f11b80c0cf0 2023-10-18T16:07:53.214 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.199+0000 7f11b6ffd700 1 --2- 172.21.15.88:0/1899913659 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f11a405f180 0x7f11a4061640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:53.214 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.199+0000 7f11b6ffd700 1 --2- 172.21.15.88:0/1899913659 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f11a405f180 0x7f11a4061640 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f11a80076c0 tx=0x7f11a800b040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:53.215 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.203+0000 7f119effd700 1 -- 172.21.15.88:0/1899913659 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f11b004b940 con 0x7f11b80c0cf0 2023-10-18T16:07:53.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.366+0000 7f11bd41d700 1 -- 172.21.15.88:0/1899913659 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f11b806c4e0 con 0x7f11b80c0cf0 2023-10-18T16:07:53.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.366+0000 7f119effd700 1 -- 172.21.15.88:0/1899913659 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f11b004ef60 con 0x7f11b80c0cf0 2023-10-18T16:07:53.370 INFO:teuthology.orchestra.run.smithi088.stdout:55834574854 2023-10-18T16:07:53.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.368+0000 7f11bd41d700 1 -- 172.21.15.88:0/1899913659 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f11a405f180 msgr2=0x7f11a4061640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:53.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.368+0000 7f11bd41d700 1 --2- 172.21.15.88:0/1899913659 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f11a405f180 0x7f11a4061640 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f11a80076c0 tx=0x7f11a800b040).stop 2023-10-18T16:07:53.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.368+0000 7f11bd41d700 1 -- 172.21.15.88:0/1899913659 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11b80c0cf0 msgr2=0x7f11b8161d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:53.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.368+0000 7f11bd41d700 1 --2- 172.21.15.88:0/1899913659 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11b80c0cf0 0x7f11b8161d20 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f11b000e8d0 tx=0x7f11b0009060).stop 2023-10-18T16:07:53.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.368+0000 7f11bd41d700 1 -- 172.21.15.88:0/1899913659 shutdown_connections 2023-10-18T16:07:53.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.368+0000 7f11bd41d700 1 --2- 172.21.15.88:0/1899913659 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f11a405f180 0x7f11a4061640 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:53.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.368+0000 7f11bd41d700 1 --2- 172.21.15.88:0/1899913659 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11b80c0cf0 0x7f11b8161d20 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:53.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.368+0000 7f11bd41d700 1 --2- 172.21.15.88:0/1899913659 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f11b80bbef0 0x7f11b80b8220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:53.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.368+0000 7f11bd41d700 1 --2- 172.21.15.88:0/1899913659 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f11b80be0f0 0x7f11b81520f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:53.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.368+0000 7f11bd41d700 1 -- 172.21.15.88:0/1899913659 >> 172.21.15.88:0/1899913659 conn(0x7f11b80c48f0 msgr2=0x7f11b80c3e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:53.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.368+0000 7f11bd41d700 1 -- 172.21.15.88:0/1899913659 shutdown_connections 2023-10-18T16:07:53.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.368+0000 7f11bd41d700 1 -- 172.21.15.88:0/1899913659 wait complete. 2023-10-18T16:07:53.384 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.6 2023-10-18T16:07:53.384 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-10-18T16:07:53.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.495+0000 7fce892e5700 1 Processor -- start 2023-10-18T16:07:53.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.495+0000 7fce892e5700 1 -- start start 2023-10-18T16:07:53.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.496+0000 7fce892e5700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fce840c4190 0x7fce84152060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:53.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.496+0000 7fce892e5700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce841518f0 con 0x7fce840c36a0 2023-10-18T16:07:53.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.496+0000 7fce892e5700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce84151bd0 con 0x7fce84154500 2023-10-18T16:07:53.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.496+0000 7fce892e5700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce8414df30 con 0x7fce840c4190 2023-10-18T16:07:53.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.496+0000 7fce837fe700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fce840c4190 0x7fce84152060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:53.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.496+0000 7fce837fe700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fce840c4190 0x7fce84152060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59728/0 (socket says 172.21.15.88:59728) 2023-10-18T16:07:53.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.496+0000 7fce82ffd700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fce840c36a0 0x7fce84064eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:52538/0 (socket says 172.21.15.88:52538) 2023-10-18T16:07:53.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.496+0000 7fce837fe700 1 -- 172.21.15.88:0/2463508754 learned_addr learned my addr 172.21.15.88:0/2463508754 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:53.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.497+0000 7fce837fe700 1 -- 172.21.15.88:0/2463508754 >> v1:172.21.15.151:6789/0 conn(0x7fce84154500 legacy=0x7fce840c3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:53.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.497+0000 7fce837fe700 1 -- 172.21.15.88:0/2463508754 >> v1:172.21.15.88:6789/0 conn(0x7fce840c36a0 legacy=0x7fce84064eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:53.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.497+0000 7fce837fe700 1 -- 172.21.15.88:0/2463508754 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce840a8bf0 con 0x7fce840c4190 2023-10-18T16:07:53.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.497+0000 7fce81ffb700 1 -- 172.21.15.88:0/2463508754 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4236312275 0 0) 0x7fce841518f0 con 0x7fce840c36a0 2023-10-18T16:07:53.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.497+0000 7fce837fe700 1 --2- 172.21.15.88:0/2463508754 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce840c4190 0x7fce84152060 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fce7c006fd0 tx=0x7fce7c0090f0).ready entity=mon.0 client_cookie=c9833ca3e7fad299 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:53.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.497+0000 7fce81ffb700 1 -- 172.21.15.88:0/2463508754 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce7c00b600 con 0x7fce840c4190 2023-10-18T16:07:53.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.497+0000 7fce81ffb700 1 -- 172.21.15.88:0/2463508754 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce7c00b760 con 0x7fce840c4190 2023-10-18T16:07:53.505 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.497+0000 7fce81ffb700 1 -- 172.21.15.88:0/2463508754 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce7c00bae0 con 0x7fce840c4190 2023-10-18T16:07:53.505 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.497+0000 7fce892e5700 1 -- 172.21.15.88:0/2463508754 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce840c4190 msgr2=0x7fce84152060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:53.505 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.497+0000 7fce892e5700 1 --2- 172.21.15.88:0/2463508754 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce840c4190 0x7fce84152060 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fce7c006fd0 tx=0x7fce7c0090f0).stop 2023-10-18T16:07:53.505 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.497+0000 7fce892e5700 1 -- 172.21.15.88:0/2463508754 shutdown_connections 2023-10-18T16:07:53.505 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.497+0000 7fce892e5700 1 --2- 172.21.15.88:0/2463508754 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce840c4190 0x7fce84152060 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:53.505 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.497+0000 7fce892e5700 1 -- 172.21.15.88:0/2463508754 >> 172.21.15.88:0/2463508754 conn(0x7fce84069d80 msgr2=0x7fce8406a190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:53.506 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.498+0000 7fce892e5700 1 -- 172.21.15.88:0/2463508754 shutdown_connections 2023-10-18T16:07:53.506 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.498+0000 7fce892e5700 1 -- 172.21.15.88:0/2463508754 wait complete. 2023-10-18T16:07:53.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.499+0000 7fce892e5700 1 Processor -- start 2023-10-18T16:07:53.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.499+0000 7fce892e5700 1 -- start start 2023-10-18T16:07:53.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.499+0000 7fce892e5700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fce840c36a0 0x7fce840c66d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:53.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.499+0000 7fce892e5700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce840c4190 0x7fce84150760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:53.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.499+0000 7fce892e5700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fce84154500 0x7fce84150c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:53.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.499+0000 7fce892e5700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce84151790 con 0x7fce840c4190 2023-10-18T16:07:53.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.499+0000 7fce892e5700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce84160050 con 0x7fce84154500 2023-10-18T16:07:53.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.499+0000 7fce892e5700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce840a6830 con 0x7fce840c36a0 2023-10-18T16:07:53.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.500+0000 7fce827fc700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce840c4190 0x7fce84150760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:53.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.500+0000 7fce827fc700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce840c4190 0x7fce84150760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59744/0 (socket says 172.21.15.88:59744) 2023-10-18T16:07:53.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.500+0000 7fce827fc700 1 -- 172.21.15.88:0/2896602340 learned_addr learned my addr 172.21.15.88:0/2896602340 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:53.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.500+0000 7fce837fe700 1 --2- 172.21.15.88:0/2896602340 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fce84154500 0x7fce84150c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:53.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.500+0000 7fce82ffd700 1 --2- 172.21.15.88:0/2896602340 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fce840c36a0 0x7fce840c66d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:53.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.500+0000 7fce827fc700 1 -- 172.21.15.88:0/2896602340 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fce840c36a0 msgr2=0x7fce840c66d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:53.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.500+0000 7fce827fc700 1 --2- 172.21.15.88:0/2896602340 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fce840c36a0 0x7fce840c66d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:53.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.500+0000 7fce827fc700 1 -- 172.21.15.88:0/2896602340 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fce84154500 msgr2=0x7fce84150c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:53.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.500+0000 7fce827fc700 1 --2- 172.21.15.88:0/2896602340 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fce84154500 0x7fce84150c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:53.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.500+0000 7fce827fc700 1 -- 172.21.15.88:0/2896602340 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce7c009370 con 0x7fce840c4190 2023-10-18T16:07:53.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.500+0000 7fce82ffd700 1 --2- 172.21.15.88:0/2896602340 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fce840c36a0 0x7fce840c66d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:07:53.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.500+0000 7fce827fc700 1 --2- 172.21.15.88:0/2896602340 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce840c4190 0x7fce84150760 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fce6c00c470 tx=0x7fce6c007e00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:53.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.500+0000 7fce7affd700 1 -- 172.21.15.88:0/2896602340 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce6c00ce30 con 0x7fce840c4190 2023-10-18T16:07:53.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.500+0000 7fce7affd700 1 -- 172.21.15.88:0/2896602340 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce6c007020 con 0x7fce840c4190 2023-10-18T16:07:53.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.500+0000 7fce892e5700 1 -- 172.21.15.88:0/2896602340 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce84160da0 con 0x7fce840c4190 2023-10-18T16:07:53.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.500+0000 7fce7affd700 1 -- 172.21.15.88:0/2896602340 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce6c0073c0 con 0x7fce840c4190 2023-10-18T16:07:53.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.500+0000 7fce892e5700 1 -- 172.21.15.88:0/2896602340 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce841605a0 con 0x7fce840c4190 2023-10-18T16:07:53.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.501+0000 7fce892e5700 1 -- 172.21.15.88:0/2896602340 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce8406c4e0 con 0x7fce840c4190 2023-10-18T16:07:53.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.501+0000 7fce7affd700 1 -- 172.21.15.88:0/2896602340 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7fce6c0068b0 con 0x7fce840c4190 2023-10-18T16:07:53.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.502+0000 7fce7affd700 1 --2- 172.21.15.88:0/2896602340 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fce6405f120 0x7fce640615e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:53.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.502+0000 7fce7affd700 1 -- 172.21.15.88:0/2896602340 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6121+0+0 (secure 0 0 0) 0x7fce6c0794d0 con 0x7fce840c4190 2023-10-18T16:07:53.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.502+0000 7fce82ffd700 1 --2- 172.21.15.88:0/2896602340 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fce6405f120 0x7fce640615e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:53.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.502+0000 7fce82ffd700 1 --2- 172.21.15.88:0/2896602340 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fce6405f120 0x7fce640615e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fce74005fd0 tx=0x7fce74007e70).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:53.515 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.506+0000 7fce7affd700 1 -- 172.21.15.88:0/2896602340 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7fce6c047ac0 con 0x7fce840c4190 2023-10-18T16:07:53.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.670+0000 7fce892e5700 1 -- 172.21.15.88:0/2896602340 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fce841529b0 con 0x7fce840c4190 2023-10-18T16:07:53.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.671+0000 7fce7affd700 1 -- 172.21.15.88:0/2896602340 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fce6c04a7a0 con 0x7fce840c4190 2023-10-18T16:07:53.675 INFO:teuthology.orchestra.run.smithi088.stdout:55834574853 2023-10-18T16:07:53.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.672+0000 7fce892e5700 1 -- 172.21.15.88:0/2896602340 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fce6405f120 msgr2=0x7fce640615e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:53.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.672+0000 7fce892e5700 1 --2- 172.21.15.88:0/2896602340 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fce6405f120 0x7fce640615e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fce74005fd0 tx=0x7fce74007e70).stop 2023-10-18T16:07:53.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.672+0000 7fce892e5700 1 -- 172.21.15.88:0/2896602340 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce840c4190 msgr2=0x7fce84150760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:53.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.672+0000 7fce892e5700 1 --2- 172.21.15.88:0/2896602340 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce840c4190 0x7fce84150760 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fce6c00c470 tx=0x7fce6c007e00).stop 2023-10-18T16:07:53.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.672+0000 7fce892e5700 1 -- 172.21.15.88:0/2896602340 shutdown_connections 2023-10-18T16:07:53.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.672+0000 7fce892e5700 1 --2- 172.21.15.88:0/2896602340 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fce6405f120 0x7fce640615e0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:53.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.672+0000 7fce892e5700 1 --2- 172.21.15.88:0/2896602340 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fce84154500 0x7fce84150c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:53.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.673+0000 7fce892e5700 1 --2- 172.21.15.88:0/2896602340 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fce840c36a0 0x7fce840c66d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:53.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.673+0000 7fce892e5700 1 --2- 172.21.15.88:0/2896602340 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce840c4190 0x7fce84150760 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:53.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.673+0000 7fce892e5700 1 -- 172.21.15.88:0/2896602340 >> 172.21.15.88:0/2896602340 conn(0x7fce84069d80 msgr2=0x7fce8414cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:53.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.673+0000 7fce892e5700 1 -- 172.21.15.88:0/2896602340 shutdown_connections 2023-10-18T16:07:53.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.673+0000 7fce892e5700 1 -- 172.21.15.88:0/2896602340 wait complete. 2023-10-18T16:07:53.688 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.7 2023-10-18T16:07:53.688 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-18T16:07:53.689 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-18T16:07:53.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.798+0000 7f43e25d3700 1 Processor -- start 2023-10-18T16:07:53.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.798+0000 7f43e25d3700 1 -- start start 2023-10-18T16:07:53.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.798+0000 7f43e25d3700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f43dc14d880 0x7f43dc154280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:53.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.799+0000 7f43e25d3700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f43dc0c1990 0x7f43dc0c1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:53.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.799+0000 7f43e25d3700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f43dc064490 0x7f43dc064920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:53.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.799+0000 7f43e25d3700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43dc0a8570 con 0x7f43dc064490 2023-10-18T16:07:53.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.799+0000 7f43e25d3700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43dc154770 con 0x7f43dc14d880 2023-10-18T16:07:53.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.799+0000 7f43e25d3700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43dc0a87c0 con 0x7f43dc0c1990 2023-10-18T16:07:53.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.799+0000 7f43e0b70700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f43dc064490 0x7f43dc064920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:53.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.799+0000 7f43e0b70700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f43dc064490 0x7f43dc064920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59754/0 (socket says 172.21.15.88:59754) 2023-10-18T16:07:53.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.799+0000 7f43e0b70700 1 -- 172.21.15.88:0/3902161729 learned_addr learned my addr 172.21.15.88:0/3902161729 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:53.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.799+0000 7f43db7fe700 1 --2- 172.21.15.88:0/3902161729 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f43dc0c1990 0x7f43dc0c1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:53.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.799+0000 7f43dbfff700 1 --2- 172.21.15.88:0/3902161729 >> v2:172.21.15.151:3300/0 conn(0x7f43dc14d880 0x7f43dc154280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:53.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.799+0000 7f43e0b70700 1 -- 172.21.15.88:0/3902161729 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f43dc0c1990 msgr2=0x7f43dc0c1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:53.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.799+0000 7f43e0b70700 1 --2- 172.21.15.88:0/3902161729 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f43dc0c1990 0x7f43dc0c1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:53.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.799+0000 7f43e0b70700 1 -- 172.21.15.88:0/3902161729 >> v2:172.21.15.151:3300/0 conn(0x7f43dc14d880 msgr2=0x7f43dc154280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:53.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.799+0000 7f43e0b70700 1 --2- 172.21.15.88:0/3902161729 >> v2:172.21.15.151:3300/0 conn(0x7f43dc14d880 0x7f43dc154280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:53.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.799+0000 7f43e0b70700 1 -- 172.21.15.88:0/3902161729 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43dc0a8aa0 con 0x7f43dc064490 2023-10-18T16:07:53.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.799+0000 7f43db7fe700 1 --2- 172.21.15.88:0/3902161729 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f43dc0c1990 0x7f43dc0c1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:07:53.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.799+0000 7f43dbfff700 1 --2- 172.21.15.88:0/3902161729 >> v2:172.21.15.151:3300/0 conn(0x7f43dc14d880 0x7f43dc154280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:07:53.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.799+0000 7f43e0b70700 1 --2- 172.21.15.88:0/3902161729 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f43dc064490 0x7f43dc064920 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f43dc062ae0 tx=0x7f43d4009840).ready entity=mon.0 client_cookie=bef5ea3409f161bf server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:53.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.799+0000 7f43daffd700 1 -- 172.21.15.88:0/3902161729 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43d400e070 con 0x7f43dc064490 2023-10-18T16:07:53.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.799+0000 7f43daffd700 1 -- 172.21.15.88:0/3902161729 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43d400ba50 con 0x7f43dc064490 2023-10-18T16:07:53.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.799+0000 7f43daffd700 1 -- 172.21.15.88:0/3902161729 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43d400bdd0 con 0x7f43dc064490 2023-10-18T16:07:53.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.800+0000 7f43e25d3700 1 -- 172.21.15.88:0/3902161729 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f43dc064490 msgr2=0x7f43dc064920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:53.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.800+0000 7f43e25d3700 1 --2- 172.21.15.88:0/3902161729 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f43dc064490 0x7f43dc064920 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f43dc062ae0 tx=0x7f43d4009840).stop 2023-10-18T16:07:53.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.800+0000 7f43e25d3700 1 -- 172.21.15.88:0/3902161729 shutdown_connections 2023-10-18T16:07:53.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.800+0000 7f43e25d3700 1 --2- 172.21.15.88:0/3902161729 >> v2:172.21.15.151:3300/0 conn(0x7f43dc14d880 0x7f43dc154280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:53.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.800+0000 7f43e25d3700 1 --2- 172.21.15.88:0/3902161729 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f43dc064490 0x7f43dc064920 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:53.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.800+0000 7f43e25d3700 1 --2- 172.21.15.88:0/3902161729 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f43dc0c1990 0x7f43dc0c1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:53.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.800+0000 7f43e25d3700 1 -- 172.21.15.88:0/3902161729 >> 172.21.15.88:0/3902161729 conn(0x7f43dc069d80 msgr2=0x7f43dc06a190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:53.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.800+0000 7f43e25d3700 1 -- 172.21.15.88:0/3902161729 shutdown_connections 2023-10-18T16:07:53.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.800+0000 7f43e25d3700 1 -- 172.21.15.88:0/3902161729 wait complete. 2023-10-18T16:07:53.813 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.801+0000 7f43e25d3700 1 Processor -- start 2023-10-18T16:07:53.813 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.801+0000 7f43e25d3700 1 -- start start 2023-10-18T16:07:53.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.802+0000 7f43e25d3700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f43dc064490 0x7f43dc149ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:53.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.802+0000 7f43e25d3700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f43dc0c1990 0x7f43dc14a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:53.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.802+0000 7f43e25d3700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f43dc14d880 0x7f43dc0ba790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:53.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.802+0000 7f43e25d3700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43dc14b250 con 0x7f43dc064490 2023-10-18T16:07:53.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.802+0000 7f43e25d3700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43dc149a10 con 0x7f43dc14d880 2023-10-18T16:07:53.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.802+0000 7f43e25d3700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43dc0bac80 con 0x7f43dc0c1990 2023-10-18T16:07:53.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.802+0000 7f43dbfff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f43dc064490 0x7f43dc149ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:53.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.802+0000 7f43dbfff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f43dc064490 0x7f43dc149ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59760/0 (socket says 172.21.15.88:59760) 2023-10-18T16:07:53.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.802+0000 7f43dbfff700 1 -- 172.21.15.88:0/448685964 learned_addr learned my addr 172.21.15.88:0/448685964 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:53.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.802+0000 7f43e0b70700 1 --2- 172.21.15.88:0/448685964 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f43dc14d880 0x7f43dc0ba790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:53.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.802+0000 7f43db7fe700 1 --2- 172.21.15.88:0/448685964 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f43dc0c1990 0x7f43dc14a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:53.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.802+0000 7f43dbfff700 1 -- 172.21.15.88:0/448685964 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f43dc0c1990 msgr2=0x7f43dc14a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:53.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.802+0000 7f43dbfff700 1 --2- 172.21.15.88:0/448685964 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f43dc0c1990 0x7f43dc14a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:53.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.802+0000 7f43dbfff700 1 -- 172.21.15.88:0/448685964 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f43dc14d880 msgr2=0x7f43dc0ba790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:53.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.802+0000 7f43dbfff700 1 --2- 172.21.15.88:0/448685964 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f43dc14d880 0x7f43dc0ba790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:53.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.802+0000 7f43dbfff700 1 -- 172.21.15.88:0/448685964 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43d4009ae0 con 0x7f43dc064490 2023-10-18T16:07:53.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.802+0000 7f43db7fe700 1 --2- 172.21.15.88:0/448685964 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f43dc0c1990 0x7f43dc14a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:07:53.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.802+0000 7f43dbfff700 1 --2- 172.21.15.88:0/448685964 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f43dc064490 0x7f43dc149ea0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f43cc00d8c0 tx=0x7f43cc008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:53.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.802+0000 7f43c3fff700 1 -- 172.21.15.88:0/448685964 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43cc0125f0 con 0x7f43dc064490 2023-10-18T16:07:53.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.802+0000 7f43c3fff700 1 -- 172.21.15.88:0/448685964 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43cc012750 con 0x7f43dc064490 2023-10-18T16:07:53.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.802+0000 7f43c3fff700 1 -- 172.21.15.88:0/448685964 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43cc012ad0 con 0x7f43dc064490 2023-10-18T16:07:53.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.803+0000 7f43e25d3700 1 -- 172.21.15.88:0/448685964 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43dc14ac90 con 0x7f43dc064490 2023-10-18T16:07:53.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.803+0000 7f43e25d3700 1 -- 172.21.15.88:0/448685964 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43dc1493c0 con 0x7f43dc064490 2023-10-18T16:07:53.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.804+0000 7f43e25d3700 1 -- 172.21.15.88:0/448685964 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43dc149220 con 0x7f43dc064490 2023-10-18T16:07:53.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.804+0000 7f43c3fff700 1 -- 172.21.15.88:0/448685964 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f43cc013480 con 0x7f43dc064490 2023-10-18T16:07:53.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.804+0000 7f43c3fff700 1 --2- 172.21.15.88:0/448685964 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f43c805f150 0x7f43c8061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:53.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.804+0000 7f43c3fff700 1 -- 172.21.15.88:0/448685964 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6121+0+0 (secure 0 0 0) 0x7f43cc07a250 con 0x7f43dc064490 2023-10-18T16:07:53.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.804+0000 7f43db7fe700 1 --2- 172.21.15.88:0/448685964 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f43c805f150 0x7f43c8061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:53.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.805+0000 7f43db7fe700 1 --2- 172.21.15.88:0/448685964 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f43c805f150 0x7f43c8061610 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f43dc0c5920 tx=0x7f43d0008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:53.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.808+0000 7f43c3fff700 1 -- 172.21.15.88:0/448685964 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f43dc149220 con 0x7f43dc064490 2023-10-18T16:07:53.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.967+0000 7f43e25d3700 1 -- 172.21.15.88:0/448685964 --> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f43dc149220 con 0x7f43c805f150 2023-10-18T16:07:53.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.977+0000 7f43c3fff700 1 -- 172.21.15.88:0/448685964 <== mgr.4102 v2:172.21.15.151:6832/90916 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+336239 (secure 0 0 0) 0x7f43dc149220 con 0x7f43c805f150 2023-10-18T16:07:53.987 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:07:53.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.980+0000 7f43e25d3700 1 -- 172.21.15.88:0/448685964 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f43c805f150 msgr2=0x7f43c8061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:53.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.980+0000 7f43e25d3700 1 --2- 172.21.15.88:0/448685964 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f43c805f150 0x7f43c8061610 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f43dc0c5920 tx=0x7f43d0008040).stop 2023-10-18T16:07:53.992 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.980+0000 7f43e25d3700 1 -- 172.21.15.88:0/448685964 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f43dc064490 msgr2=0x7f43dc149ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:53.992 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.980+0000 7f43e25d3700 1 --2- 172.21.15.88:0/448685964 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f43dc064490 0x7f43dc149ea0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f43cc00d8c0 tx=0x7f43cc008040).stop 2023-10-18T16:07:53.992 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.980+0000 7f43e25d3700 1 -- 172.21.15.88:0/448685964 shutdown_connections 2023-10-18T16:07:53.992 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.980+0000 7f43e25d3700 1 --2- 172.21.15.88:0/448685964 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f43c805f150 0x7f43c8061610 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:53.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.980+0000 7f43e25d3700 1 --2- 172.21.15.88:0/448685964 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f43dc14d880 0x7f43dc0ba790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:53.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.980+0000 7f43e25d3700 1 --2- 172.21.15.88:0/448685964 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f43dc064490 0x7f43dc149ea0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:53.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.980+0000 7f43e25d3700 1 --2- 172.21.15.88:0/448685964 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f43dc0c1990 0x7f43dc14a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:53.994 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.981+0000 7f43e25d3700 1 -- 172.21.15.88:0/448685964 >> 172.21.15.88:0/448685964 conn(0x7f43dc069d80 msgr2=0x7f43dc1564b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:53.994 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.981+0000 7f43e25d3700 1 -- 172.21.15.88:0/448685964 shutdown_connections 2023-10-18T16:07:53.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:53.981+0000 7f43e25d3700 1 -- 172.21.15.88:0/448685964 wait complete. 2023-10-18T16:07:53.995 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2023-10-18T16:07:53.996 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2023-10-18T16:07:52.459463+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":274,"acting":274,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":274,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2378560,"kb_used_data":1792,"kb_used_omap":0,"kb_used_meta":2376704,"kb_avail":752596160,"statfs":{"total":773094113280,"available":770658467840,"internally_reserved":0,"allocated":1835008,"data_stored":682728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2433744896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"4.1d","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.800856+0000","last_change":"2023-10-18T16:07:41.796671+0000","last_active":"2023-10-18T16:07:41.800856+0000","last_peered":"2023-10-18T16:07:41.800856+0000","last_clean":"2023-10-18T16:07:41.800856+0000","last_became_active":"2023-10-18T16:07:41.796451+0000","last_became_peered":"2023-10-18T16:07:41.796451+0000","last_unstale":"2023-10-18T16:07:41.800856+0000","last_undegraded":"2023-10-18T16:07:41.800856+0000","last_fullsized":"2023-10-18T16:07:41.800856+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.794828+0000","last_change":"2023-10-18T16:07:41.442111+0000","last_active":"2023-10-18T16:07:41.794828+0000","last_peered":"2023-10-18T16:07:41.794828+0000","last_clean":"2023-10-18T16:07:41.794828+0000","last_became_active":"2023-10-18T16:07:41.441815+0000","last_became_peered":"2023-10-18T16:07:41.441815+0000","last_unstale":"2023-10-18T16:07:41.794828+0000","last_undegraded":"2023-10-18T16:07:41.794828+0000","last_fullsized":"2023-10-18T16:07:41.794828+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.809204+0000","last_change":"2023-10-18T16:07:41.808517+0000","last_active":"2023-10-18T16:07:41.809204+0000","last_peered":"2023-10-18T16:07:41.809204+0000","last_clean":"2023-10-18T16:07:41.809204+0000","last_became_active":"2023-10-18T16:07:41.808224+0000","last_became_peered":"2023-10-18T16:07:41.808224+0000","last_unstale":"2023-10-18T16:07:41.809204+0000","last_undegraded":"2023-10-18T16:07:41.809204+0000","last_fullsized":"2023-10-18T16:07:41.809204+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.785683+0000","last_change":"2023-10-18T16:07:41.430021+0000","last_active":"2023-10-18T16:07:41.785683+0000","last_peered":"2023-10-18T16:07:41.785683+0000","last_clean":"2023-10-18T16:07:41.785683+0000","last_became_active":"2023-10-18T16:07:41.429781+0000","last_became_peered":"2023-10-18T16:07:41.429781+0000","last_unstale":"2023-10-18T16:07:41.785683+0000","last_undegraded":"2023-10-18T16:07:41.785683+0000","last_fullsized":"2023-10-18T16:07:41.785683+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.808894+0000","last_change":"2023-10-18T16:07:41.807746+0000","last_active":"2023-10-18T16:07:41.808894+0000","last_peered":"2023-10-18T16:07:41.808894+0000","last_clean":"2023-10-18T16:07:41.808894+0000","last_became_active":"2023-10-18T16:07:41.807256+0000","last_became_peered":"2023-10-18T16:07:41.807256+0000","last_unstale":"2023-10-18T16:07:41.808894+0000","last_undegraded":"2023-10-18T16:07:41.808894+0000","last_fullsized":"2023-10-18T16:07:41.808894+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.788944+0000","last_change":"2023-10-18T16:07:41.431122+0000","last_active":"2023-10-18T16:07:41.788944+0000","last_peered":"2023-10-18T16:07:41.788944+0000","last_clean":"2023-10-18T16:07:41.788944+0000","last_became_active":"2023-10-18T16:07:41.430831+0000","last_became_peered":"2023-10-18T16:07:41.430831+0000","last_unstale":"2023-10-18T16:07:41.788944+0000","last_undegraded":"2023-10-18T16:07:41.788944+0000","last_fullsized":"2023-10-18T16:07:41.788944+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.807269+0000","last_change":"2023-10-18T16:07:41.801866+0000","last_active":"2023-10-18T16:07:41.807269+0000","last_peered":"2023-10-18T16:07:41.807269+0000","last_clean":"2023-10-18T16:07:41.807269+0000","last_became_active":"2023-10-18T16:07:41.801602+0000","last_became_peered":"2023-10-18T16:07:41.801602+0000","last_unstale":"2023-10-18T16:07:41.807269+0000","last_undegraded":"2023-10-18T16:07:41.807269+0000","last_fullsized":"2023-10-18T16:07:41.807269+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.790519+0000","last_change":"2023-10-18T16:07:41.442448+0000","last_active":"2023-10-18T16:07:41.790519+0000","last_peered":"2023-10-18T16:07:41.790519+0000","last_clean":"2023-10-18T16:07:41.790519+0000","last_became_active":"2023-10-18T16:07:41.442143+0000","last_became_peered":"2023-10-18T16:07:41.442143+0000","last_unstale":"2023-10-18T16:07:41.790519+0000","last_undegraded":"2023-10-18T16:07:41.790519+0000","last_fullsized":"2023-10-18T16:07:41.790519+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.808841+0000","last_change":"2023-10-18T16:07:41.808302+0000","last_active":"2023-10-18T16:07:41.808841+0000","last_peered":"2023-10-18T16:07:41.808841+0000","last_clean":"2023-10-18T16:07:41.808841+0000","last_became_active":"2023-10-18T16:07:41.808001+0000","last_became_peered":"2023-10-18T16:07:41.808001+0000","last_unstale":"2023-10-18T16:07:41.808841+0000","last_undegraded":"2023-10-18T16:07:41.808841+0000","last_fullsized":"2023-10-18T16:07:41.808841+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.790557+0000","last_change":"2023-10-18T16:07:41.433080+0000","last_active":"2023-10-18T16:07:41.790557+0000","last_peered":"2023-10-18T16:07:41.790557+0000","last_clean":"2023-10-18T16:07:41.790557+0000","last_became_active":"2023-10-18T16:07:41.432714+0000","last_became_peered":"2023-10-18T16:07:41.432714+0000","last_unstale":"2023-10-18T16:07:41.790557+0000","last_undegraded":"2023-10-18T16:07:41.790557+0000","last_fullsized":"2023-10-18T16:07:41.790557+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.803707+0000","last_change":"2023-10-18T16:07:41.797599+0000","last_active":"2023-10-18T16:07:41.803707+0000","last_peered":"2023-10-18T16:07:41.803707+0000","last_clean":"2023-10-18T16:07:41.803707+0000","last_became_active":"2023-10-18T16:07:41.797318+0000","last_became_peered":"2023-10-18T16:07:41.797318+0000","last_unstale":"2023-10-18T16:07:41.803707+0000","last_undegraded":"2023-10-18T16:07:41.803707+0000","last_fullsized":"2023-10-18T16:07:41.803707+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.798325+0000","last_change":"2023-10-18T16:07:41.437425+0000","last_active":"2023-10-18T16:07:41.798325+0000","last_peered":"2023-10-18T16:07:41.798325+0000","last_clean":"2023-10-18T16:07:41.798325+0000","last_became_active":"2023-10-18T16:07:41.437135+0000","last_became_peered":"2023-10-18T16:07:41.437135+0000","last_unstale":"2023-10-18T16:07:41.798325+0000","last_undegraded":"2023-10-18T16:07:41.798325+0000","last_fullsized":"2023-10-18T16:07:41.798325+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.806231+0000","last_change":"2023-10-18T16:07:41.802908+0000","last_active":"2023-10-18T16:07:41.806231+0000","last_peered":"2023-10-18T16:07:41.806231+0000","last_clean":"2023-10-18T16:07:41.806231+0000","last_became_active":"2023-10-18T16:07:41.802644+0000","last_became_peered":"2023-10-18T16:07:41.802644+0000","last_unstale":"2023-10-18T16:07:41.806231+0000","last_undegraded":"2023-10-18T16:07:41.806231+0000","last_fullsized":"2023-10-18T16:07:41.806231+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.796534+0000","last_change":"2023-10-18T16:07:41.446842+0000","last_active":"2023-10-18T16:07:41.796534+0000","last_peered":"2023-10-18T16:07:41.796534+0000","last_clean":"2023-10-18T16:07:41.796534+0000","last_became_active":"2023-10-18T16:07:41.446567+0000","last_became_peered":"2023-10-18T16:07:41.446567+0000","last_unstale":"2023-10-18T16:07:41.796534+0000","last_undegraded":"2023-10-18T16:07:41.796534+0000","last_fullsized":"2023-10-18T16:07:41.796534+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.802559+0000","last_change":"2023-10-18T16:07:41.798736+0000","last_active":"2023-10-18T16:07:41.802559+0000","last_peered":"2023-10-18T16:07:41.802559+0000","last_clean":"2023-10-18T16:07:41.802559+0000","last_became_active":"2023-10-18T16:07:41.798458+0000","last_became_peered":"2023-10-18T16:07:41.798458+0000","last_unstale":"2023-10-18T16:07:41.802559+0000","last_undegraded":"2023-10-18T16:07:41.802559+0000","last_fullsized":"2023-10-18T16:07:41.802559+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.796971+0000","last_change":"2023-10-18T16:07:41.443369+0000","last_active":"2023-10-18T16:07:41.796971+0000","last_peered":"2023-10-18T16:07:41.796971+0000","last_clean":"2023-10-18T16:07:41.796971+0000","last_became_active":"2023-10-18T16:07:41.443064+0000","last_became_peered":"2023-10-18T16:07:41.443064+0000","last_unstale":"2023-10-18T16:07:41.796971+0000","last_undegraded":"2023-10-18T16:07:41.796971+0000","last_fullsized":"2023-10-18T16:07:41.796971+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.809725+0000","last_change":"2023-10-18T16:07:41.808196+0000","last_active":"2023-10-18T16:07:41.809725+0000","last_peered":"2023-10-18T16:07:41.809725+0000","last_clean":"2023-10-18T16:07:41.809725+0000","last_became_active":"2023-10-18T16:07:41.807537+0000","last_became_peered":"2023-10-18T16:07:41.807537+0000","last_unstale":"2023-10-18T16:07:41.809725+0000","last_undegraded":"2023-10-18T16:07:41.809725+0000","last_fullsized":"2023-10-18T16:07:41.809725+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.791192+0000","last_change":"2023-10-18T16:07:41.433498+0000","last_active":"2023-10-18T16:07:41.791192+0000","last_peered":"2023-10-18T16:07:41.791192+0000","last_clean":"2023-10-18T16:07:41.791192+0000","last_became_active":"2023-10-18T16:07:41.433321+0000","last_became_peered":"2023-10-18T16:07:41.433321+0000","last_unstale":"2023-10-18T16:07:41.791192+0000","last_undegraded":"2023-10-18T16:07:41.791192+0000","last_fullsized":"2023-10-18T16:07:41.791192+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.801992+0000","last_change":"2023-10-18T16:07:41.797372+0000","last_active":"2023-10-18T16:07:41.801992+0000","last_peered":"2023-10-18T16:07:41.801992+0000","last_clean":"2023-10-18T16:07:41.801992+0000","last_became_active":"2023-10-18T16:07:41.797108+0000","last_became_peered":"2023-10-18T16:07:41.797108+0000","last_unstale":"2023-10-18T16:07:41.801992+0000","last_undegraded":"2023-10-18T16:07:41.801992+0000","last_fullsized":"2023-10-18T16:07:41.801992+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.799678+0000","last_change":"2023-10-18T16:07:41.445469+0000","last_active":"2023-10-18T16:07:41.799678+0000","last_peered":"2023-10-18T16:07:41.799678+0000","last_clean":"2023-10-18T16:07:41.799678+0000","last_became_active":"2023-10-18T16:07:41.445204+0000","last_became_peered":"2023-10-18T16:07:41.445204+0000","last_unstale":"2023-10-18T16:07:41.799678+0000","last_undegraded":"2023-10-18T16:07:41.799678+0000","last_fullsized":"2023-10-18T16:07:41.799678+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.808614+0000","last_change":"2023-10-18T16:07:41.808046+0000","last_active":"2023-10-18T16:07:41.808614+0000","last_peered":"2023-10-18T16:07:41.808614+0000","last_clean":"2023-10-18T16:07:41.808614+0000","last_became_active":"2023-10-18T16:07:41.807189+0000","last_became_peered":"2023-10-18T16:07:41.807189+0000","last_unstale":"2023-10-18T16:07:41.808614+0000","last_undegraded":"2023-10-18T16:07:41.808614+0000","last_fullsized":"2023-10-18T16:07:41.808614+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.792937+0000","last_change":"2023-10-18T16:07:41.434313+0000","last_active":"2023-10-18T16:07:41.792937+0000","last_peered":"2023-10-18T16:07:41.792937+0000","last_clean":"2023-10-18T16:07:41.792937+0000","last_became_active":"2023-10-18T16:07:41.434022+0000","last_became_peered":"2023-10-18T16:07:41.434022+0000","last_unstale":"2023-10-18T16:07:41.792937+0000","last_undegraded":"2023-10-18T16:07:41.792937+0000","last_fullsized":"2023-10-18T16:07:41.792937+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.806794+0000","last_change":"2023-10-18T16:07:41.800593+0000","last_active":"2023-10-18T16:07:41.806794+0000","last_peered":"2023-10-18T16:07:41.806794+0000","last_clean":"2023-10-18T16:07:41.806794+0000","last_became_active":"2023-10-18T16:07:41.800302+0000","last_became_peered":"2023-10-18T16:07:41.800302+0000","last_unstale":"2023-10-18T16:07:41.806794+0000","last_undegraded":"2023-10-18T16:07:41.806794+0000","last_fullsized":"2023-10-18T16:07:41.806794+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.798209+0000","last_change":"2023-10-18T16:07:41.442977+0000","last_active":"2023-10-18T16:07:41.798209+0000","last_peered":"2023-10-18T16:07:41.798209+0000","last_clean":"2023-10-18T16:07:41.798209+0000","last_became_active":"2023-10-18T16:07:41.442383+0000","last_became_peered":"2023-10-18T16:07:41.442383+0000","last_unstale":"2023-10-18T16:07:41.798209+0000","last_undegraded":"2023-10-18T16:07:41.798209+0000","last_fullsized":"2023-10-18T16:07:41.798209+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.810814+0000","last_change":"2023-10-18T16:07:41.808758+0000","last_active":"2023-10-18T16:07:41.810814+0000","last_peered":"2023-10-18T16:07:41.810814+0000","last_clean":"2023-10-18T16:07:41.810814+0000","last_became_active":"2023-10-18T16:07:41.808475+0000","last_became_peered":"2023-10-18T16:07:41.808475+0000","last_unstale":"2023-10-18T16:07:41.810814+0000","last_undegraded":"2023-10-18T16:07:41.810814+0000","last_fullsized":"2023-10-18T16:07:41.810814+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.798959+0000","last_change":"2023-10-18T16:07:41.442566+0000","last_active":"2023-10-18T16:07:41.798959+0000","last_peered":"2023-10-18T16:07:41.798959+0000","last_clean":"2023-10-18T16:07:41.798959+0000","last_became_active":"2023-10-18T16:07:41.442264+0000","last_became_peered":"2023-10-18T16:07:41.442264+0000","last_unstale":"2023-10-18T16:07:41.798959+0000","last_undegraded":"2023-10-18T16:07:41.798959+0000","last_fullsized":"2023-10-18T16:07:41.798959+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.805558+0000","last_change":"2023-10-18T16:07:41.803408+0000","last_active":"2023-10-18T16:07:41.805558+0000","last_peered":"2023-10-18T16:07:41.805558+0000","last_clean":"2023-10-18T16:07:41.805558+0000","last_became_active":"2023-10-18T16:07:41.803142+0000","last_became_peered":"2023-10-18T16:07:41.803142+0000","last_unstale":"2023-10-18T16:07:41.805558+0000","last_undegraded":"2023-10-18T16:07:41.805558+0000","last_fullsized":"2023-10-18T16:07:41.805558+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.799082+0000","last_change":"2023-10-18T16:07:41.440460+0000","last_active":"2023-10-18T16:07:41.799082+0000","last_peered":"2023-10-18T16:07:41.799082+0000","last_clean":"2023-10-18T16:07:41.799082+0000","last_became_active":"2023-10-18T16:07:41.440162+0000","last_became_peered":"2023-10-18T16:07:41.440162+0000","last_unstale":"2023-10-18T16:07:41.799082+0000","last_undegraded":"2023-10-18T16:07:41.799082+0000","last_fullsized":"2023-10-18T16:07:41.799082+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.807426+0000","last_change":"2023-10-18T16:07:41.804730+0000","last_active":"2023-10-18T16:07:41.807426+0000","last_peered":"2023-10-18T16:07:41.807426+0000","last_clean":"2023-10-18T16:07:41.807426+0000","last_became_active":"2023-10-18T16:07:41.804455+0000","last_became_peered":"2023-10-18T16:07:41.804455+0000","last_unstale":"2023-10-18T16:07:41.807426+0000","last_undegraded":"2023-10-18T16:07:41.807426+0000","last_fullsized":"2023-10-18T16:07:41.807426+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.790384+0000","last_change":"2023-10-18T16:07:41.429156+0000","last_active":"2023-10-18T16:07:41.790384+0000","last_peered":"2023-10-18T16:07:41.790384+0000","last_clean":"2023-10-18T16:07:41.790384+0000","last_became_active":"2023-10-18T16:07:41.428828+0000","last_became_peered":"2023-10-18T16:07:41.428828+0000","last_unstale":"2023-10-18T16:07:41.790384+0000","last_undegraded":"2023-10-18T16:07:41.790384+0000","last_fullsized":"2023-10-18T16:07:41.790384+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.811080+0000","last_change":"2023-10-18T16:07:41.810261+0000","last_active":"2023-10-18T16:07:41.811080+0000","last_peered":"2023-10-18T16:07:41.811080+0000","last_clean":"2023-10-18T16:07:41.811080+0000","last_became_active":"2023-10-18T16:07:41.809938+0000","last_became_peered":"2023-10-18T16:07:41.809938+0000","last_unstale":"2023-10-18T16:07:41.811080+0000","last_undegraded":"2023-10-18T16:07:41.811080+0000","last_fullsized":"2023-10-18T16:07:41.811080+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.788799+0000","last_change":"2023-10-18T16:07:41.436527+0000","last_active":"2023-10-18T16:07:41.788799+0000","last_peered":"2023-10-18T16:07:41.788799+0000","last_clean":"2023-10-18T16:07:41.788799+0000","last_became_active":"2023-10-18T16:07:41.436223+0000","last_became_peered":"2023-10-18T16:07:41.436223+0000","last_unstale":"2023-10-18T16:07:41.788799+0000","last_undegraded":"2023-10-18T16:07:41.788799+0000","last_fullsized":"2023-10-18T16:07:41.788799+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.806186+0000","last_change":"2023-10-18T16:07:41.802220+0000","last_active":"2023-10-18T16:07:41.806186+0000","last_peered":"2023-10-18T16:07:41.806186+0000","last_clean":"2023-10-18T16:07:41.806186+0000","last_became_active":"2023-10-18T16:07:41.801909+0000","last_became_peered":"2023-10-18T16:07:41.801909+0000","last_unstale":"2023-10-18T16:07:41.806186+0000","last_undegraded":"2023-10-18T16:07:41.806186+0000","last_fullsized":"2023-10-18T16:07:41.806186+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.793937+0000","last_change":"2023-10-18T16:07:41.446075+0000","last_active":"2023-10-18T16:07:41.793937+0000","last_peered":"2023-10-18T16:07:41.793937+0000","last_clean":"2023-10-18T16:07:41.793937+0000","last_became_active":"2023-10-18T16:07:41.445817+0000","last_became_peered":"2023-10-18T16:07:41.445817+0000","last_unstale":"2023-10-18T16:07:41.793937+0000","last_undegraded":"2023-10-18T16:07:41.793937+0000","last_fullsized":"2023-10-18T16:07:41.793937+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.806073+0000","last_change":"2023-10-18T16:07:41.802289+0000","last_active":"2023-10-18T16:07:41.806073+0000","last_peered":"2023-10-18T16:07:41.806073+0000","last_clean":"2023-10-18T16:07:41.806073+0000","last_became_active":"2023-10-18T16:07:41.802038+0000","last_became_peered":"2023-10-18T16:07:41.802038+0000","last_unstale":"2023-10-18T16:07:41.806073+0000","last_undegraded":"2023-10-18T16:07:41.806073+0000","last_fullsized":"2023-10-18T16:07:41.806073+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.795284+0000","last_change":"2023-10-18T16:07:41.445650+0000","last_active":"2023-10-18T16:07:41.795284+0000","last_peered":"2023-10-18T16:07:41.795284+0000","last_clean":"2023-10-18T16:07:41.795284+0000","last_became_active":"2023-10-18T16:07:41.445297+0000","last_became_peered":"2023-10-18T16:07:41.445297+0000","last_unstale":"2023-10-18T16:07:41.795284+0000","last_undegraded":"2023-10-18T16:07:41.795284+0000","last_fullsized":"2023-10-18T16:07:41.795284+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.807219+0000","last_change":"2023-10-18T16:07:41.800350+0000","last_active":"2023-10-18T16:07:41.807219+0000","last_peered":"2023-10-18T16:07:41.807219+0000","last_clean":"2023-10-18T16:07:41.807219+0000","last_became_active":"2023-10-18T16:07:41.800095+0000","last_became_peered":"2023-10-18T16:07:41.800095+0000","last_unstale":"2023-10-18T16:07:41.807219+0000","last_undegraded":"2023-10-18T16:07:41.807219+0000","last_fullsized":"2023-10-18T16:07:41.807219+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.789807+0000","last_change":"2023-10-18T16:07:41.429381+0000","last_active":"2023-10-18T16:07:41.789807+0000","last_peered":"2023-10-18T16:07:41.789807+0000","last_clean":"2023-10-18T16:07:41.789807+0000","last_became_active":"2023-10-18T16:07:41.429117+0000","last_became_peered":"2023-10-18T16:07:41.429117+0000","last_unstale":"2023-10-18T16:07:41.789807+0000","last_undegraded":"2023-10-18T16:07:41.789807+0000","last_fullsized":"2023-10-18T16:07:41.789807+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.786761+0000","last_change":"2023-10-18T16:07:37.254194+0000","last_active":"2023-10-18T16:07:41.786761+0000","last_peered":"2023-10-18T16:07:41.786761+0000","last_clean":"2023-10-18T16:07:41.786761+0000","last_became_active":"2023-10-18T16:07:37.253900+0000","last_became_peered":"2023-10-18T16:07:37.253900+0000","last_unstale":"2023-10-18T16:07:41.786761+0000","last_undegraded":"2023-10-18T16:07:41.786761+0000","last_fullsized":"2023-10-18T16:07:41.786761+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:35.522314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:35.522314+0000","last_clean_scrub_stamp":"2023-10-18T16:07:35.522314+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.790385+0000","last_change":"2023-10-18T16:07:41.436498+0000","last_active":"2023-10-18T16:07:41.790385+0000","last_peered":"2023-10-18T16:07:41.790385+0000","last_clean":"2023-10-18T16:07:41.790385+0000","last_became_active":"2023-10-18T16:07:41.436182+0000","last_became_peered":"2023-10-18T16:07:41.436182+0000","last_unstale":"2023-10-18T16:07:41.790385+0000","last_undegraded":"2023-10-18T16:07:41.790385+0000","last_fullsized":"2023-10-18T16:07:41.790385+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.800030+0000","last_change":"2023-10-18T16:07:41.793823+0000","last_active":"2023-10-18T16:07:41.800030+0000","last_peered":"2023-10-18T16:07:41.800030+0000","last_clean":"2023-10-18T16:07:41.800030+0000","last_became_active":"2023-10-18T16:07:41.792600+0000","last_became_peered":"2023-10-18T16:07:41.792600+0000","last_unstale":"2023-10-18T16:07:41.800030+0000","last_undegraded":"2023-10-18T16:07:41.800030+0000","last_fullsized":"2023-10-18T16:07:41.800030+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.790199+0000","last_change":"2023-10-18T16:07:36.544503+0000","last_active":"2023-10-18T16:07:41.790199+0000","last_peered":"2023-10-18T16:07:41.790199+0000","last_clean":"2023-10-18T16:07:41.790199+0000","last_became_active":"2023-10-18T16:07:36.544254+0000","last_became_peered":"2023-10-18T16:07:36.544254+0000","last_unstale":"2023-10-18T16:07:41.790199+0000","last_undegraded":"2023-10-18T16:07:41.790199+0000","last_fullsized":"2023-10-18T16:07:41.790199+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:35.522314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:35.522314+0000","last_clean_scrub_stamp":"2023-10-18T16:07:35.522314+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.795162+0000","last_change":"2023-10-18T16:07:41.437532+0000","last_active":"2023-10-18T16:07:41.795162+0000","last_peered":"2023-10-18T16:07:41.795162+0000","last_clean":"2023-10-18T16:07:41.795162+0000","last_became_active":"2023-10-18T16:07:41.437257+0000","last_became_peered":"2023-10-18T16:07:41.437257+0000","last_unstale":"2023-10-18T16:07:41.795162+0000","last_undegraded":"2023-10-18T16:07:41.795162+0000","last_fullsized":"2023-10-18T16:07:41.795162+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.807397+0000","last_change":"2023-10-18T16:07:41.801719+0000","last_active":"2023-10-18T16:07:41.807397+0000","last_peered":"2023-10-18T16:07:41.807397+0000","last_clean":"2023-10-18T16:07:41.807397+0000","last_became_active":"2023-10-18T16:07:41.801325+0000","last_became_peered":"2023-10-18T16:07:41.801325+0000","last_unstale":"2023-10-18T16:07:41.807397+0000","last_undegraded":"2023-10-18T16:07:41.807397+0000","last_fullsized":"2023-10-18T16:07:41.807397+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.799483+0000","last_change":"2023-10-18T16:07:41.791251+0000","last_active":"2023-10-18T16:07:41.799483+0000","last_peered":"2023-10-18T16:07:41.799483+0000","last_clean":"2023-10-18T16:07:41.799483+0000","last_became_active":"2023-10-18T16:07:41.790992+0000","last_became_peered":"2023-10-18T16:07:41.790992+0000","last_unstale":"2023-10-18T16:07:41.799483+0000","last_undegraded":"2023-10-18T16:07:41.799483+0000","last_fullsized":"2023-10-18T16:07:41.799483+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.795059+0000","last_change":"2023-10-18T16:07:36.542401+0000","last_active":"2023-10-18T16:07:41.795059+0000","last_peered":"2023-10-18T16:07:41.795059+0000","last_clean":"2023-10-18T16:07:41.795059+0000","last_became_active":"2023-10-18T16:07:36.542071+0000","last_became_peered":"2023-10-18T16:07:36.542071+0000","last_unstale":"2023-10-18T16:07:41.795059+0000","last_undegraded":"2023-10-18T16:07:41.795059+0000","last_fullsized":"2023-10-18T16:07:41.795059+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:35.522314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:35.522314+0000","last_clean_scrub_stamp":"2023-10-18T16:07:35.522314+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.785696+0000","last_change":"2023-10-18T16:07:41.428730+0000","last_active":"2023-10-18T16:07:41.785696+0000","last_peered":"2023-10-18T16:07:41.785696+0000","last_clean":"2023-10-18T16:07:41.785696+0000","last_became_active":"2023-10-18T16:07:41.428469+0000","last_became_peered":"2023-10-18T16:07:41.428469+0000","last_unstale":"2023-10-18T16:07:41.785696+0000","last_undegraded":"2023-10-18T16:07:41.785696+0000","last_fullsized":"2023-10-18T16:07:41.785696+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.802624+0000","last_change":"2023-10-18T16:07:41.798241+0000","last_active":"2023-10-18T16:07:41.802624+0000","last_peered":"2023-10-18T16:07:41.802624+0000","last_clean":"2023-10-18T16:07:41.802624+0000","last_became_active":"2023-10-18T16:07:41.797958+0000","last_became_peered":"2023-10-18T16:07:41.797958+0000","last_unstale":"2023-10-18T16:07:41.802624+0000","last_undegraded":"2023-10-18T16:07:41.802624+0000","last_fullsized":"2023-10-18T16:07:41.802624+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.790123+0000","last_change":"2023-10-18T16:07:36.542651+0000","last_active":"2023-10-18T16:07:41.790123+0000","last_peered":"2023-10-18T16:07:41.790123+0000","last_clean":"2023-10-18T16:07:41.790123+0000","last_became_active":"2023-10-18T16:07:36.542306+0000","last_became_peered":"2023-10-18T16:07:36.542306+0000","last_unstale":"2023-10-18T16:07:41.790123+0000","last_undegraded":"2023-10-18T16:07:41.790123+0000","last_fullsized":"2023-10-18T16:07:41.790123+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:35.522314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:35.522314+0000","last_clean_scrub_stamp":"2023-10-18T16:07:35.522314+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.795394+0000","last_change":"2023-10-18T16:07:34.567589+0000","last_active":"2023-10-18T16:07:41.795394+0000","last_peered":"2023-10-18T16:07:41.795394+0000","last_clean":"2023-10-18T16:07:41.795394+0000","last_became_active":"2023-10-18T16:07:34.567301+0000","last_became_peered":"2023-10-18T16:07:34.567301+0000","last_unstale":"2023-10-18T16:07:41.795394+0000","last_undegraded":"2023-10-18T16:07:41.795394+0000","last_fullsized":"2023-10-18T16:07:41.795394+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:33.506699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:33.506699+0000","last_clean_scrub_stamp":"2023-10-18T16:07:33.506699+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.788528+0000","last_change":"2023-10-18T16:07:41.438974+0000","last_active":"2023-10-18T16:07:41.788528+0000","last_peered":"2023-10-18T16:07:41.788528+0000","last_clean":"2023-10-18T16:07:41.788528+0000","last_became_active":"2023-10-18T16:07:41.438710+0000","last_became_peered":"2023-10-18T16:07:41.438710+0000","last_unstale":"2023-10-18T16:07:41.788528+0000","last_undegraded":"2023-10-18T16:07:41.788528+0000","last_fullsized":"2023-10-18T16:07:41.788528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.805329+0000","last_change":"2023-10-18T16:07:41.800880+0000","last_active":"2023-10-18T16:07:41.805329+0000","last_peered":"2023-10-18T16:07:41.805329+0000","last_clean":"2023-10-18T16:07:41.805329+0000","last_became_active":"2023-10-18T16:07:41.800478+0000","last_became_peered":"2023-10-18T16:07:41.800478+0000","last_unstale":"2023-10-18T16:07:41.805329+0000","last_undegraded":"2023-10-18T16:07:41.805329+0000","last_fullsized":"2023-10-18T16:07:41.805329+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.795831+0000","last_change":"2023-10-18T16:07:36.542454+0000","last_active":"2023-10-18T16:07:41.795831+0000","last_peered":"2023-10-18T16:07:41.795831+0000","last_clean":"2023-10-18T16:07:41.795831+0000","last_became_active":"2023-10-18T16:07:36.542134+0000","last_became_peered":"2023-10-18T16:07:36.542134+0000","last_unstale":"2023-10-18T16:07:41.795831+0000","last_undegraded":"2023-10-18T16:07:41.795831+0000","last_fullsized":"2023-10-18T16:07:41.795831+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:35.522314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:35.522314+0000","last_clean_scrub_stamp":"2023-10-18T16:07:35.522314+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.790424+0000","last_change":"2023-10-18T16:07:41.445653+0000","last_active":"2023-10-18T16:07:41.790424+0000","last_peered":"2023-10-18T16:07:41.790424+0000","last_clean":"2023-10-18T16:07:41.790424+0000","last_became_active":"2023-10-18T16:07:41.445399+0000","last_became_peered":"2023-10-18T16:07:41.445399+0000","last_unstale":"2023-10-18T16:07:41.790424+0000","last_undegraded":"2023-10-18T16:07:41.790424+0000","last_fullsized":"2023-10-18T16:07:41.790424+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.798965+0000","last_change":"2023-10-18T16:07:37.254490+0000","last_active":"2023-10-18T16:07:41.798965+0000","last_peered":"2023-10-18T16:07:41.798965+0000","last_clean":"2023-10-18T16:07:41.798965+0000","last_became_active":"2023-10-18T16:07:37.254182+0000","last_became_peered":"2023-10-18T16:07:37.254182+0000","last_unstale":"2023-10-18T16:07:41.798965+0000","last_undegraded":"2023-10-18T16:07:41.798965+0000","last_fullsized":"2023-10-18T16:07:41.798965+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:35.522314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:35.522314+0000","last_clean_scrub_stamp":"2023-10-18T16:07:35.522314+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.787199+0000","last_change":"2023-10-18T16:07:41.428721+0000","last_active":"2023-10-18T16:07:41.787199+0000","last_peered":"2023-10-18T16:07:41.787199+0000","last_clean":"2023-10-18T16:07:41.787199+0000","last_became_active":"2023-10-18T16:07:41.428460+0000","last_became_peered":"2023-10-18T16:07:41.428460+0000","last_unstale":"2023-10-18T16:07:41.787199+0000","last_undegraded":"2023-10-18T16:07:41.787199+0000","last_fullsized":"2023-10-18T16:07:41.787199+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.805475+0000","last_change":"2023-10-18T16:07:41.801499+0000","last_active":"2023-10-18T16:07:41.805475+0000","last_peered":"2023-10-18T16:07:41.805475+0000","last_clean":"2023-10-18T16:07:41.805475+0000","last_became_active":"2023-10-18T16:07:41.800625+0000","last_became_peered":"2023-10-18T16:07:41.800625+0000","last_unstale":"2023-10-18T16:07:41.805475+0000","last_undegraded":"2023-10-18T16:07:41.805475+0000","last_fullsized":"2023-10-18T16:07:41.805475+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.789569+0000","last_change":"2023-10-18T16:07:41.435428+0000","last_active":"2023-10-18T16:07:41.789569+0000","last_peered":"2023-10-18T16:07:41.789569+0000","last_clean":"2023-10-18T16:07:41.789569+0000","last_became_active":"2023-10-18T16:07:41.435124+0000","last_became_peered":"2023-10-18T16:07:41.435124+0000","last_unstale":"2023-10-18T16:07:41.789569+0000","last_undegraded":"2023-10-18T16:07:41.789569+0000","last_fullsized":"2023-10-18T16:07:41.789569+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.785797+0000","last_change":"2023-10-18T16:07:36.542463+0000","last_active":"2023-10-18T16:07:41.785797+0000","last_peered":"2023-10-18T16:07:41.785797+0000","last_clean":"2023-10-18T16:07:41.785797+0000","last_became_active":"2023-10-18T16:07:36.542090+0000","last_became_peered":"2023-10-18T16:07:36.542090+0000","last_unstale":"2023-10-18T16:07:41.785797+0000","last_undegraded":"2023-10-18T16:07:41.785797+0000","last_fullsized":"2023-10-18T16:07:41.785797+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:35.522314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:35.522314+0000","last_clean_scrub_stamp":"2023-10-18T16:07:35.522314+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.807288+0000","last_change":"2023-10-18T16:07:41.802294+0000","last_active":"2023-10-18T16:07:41.807288+0000","last_peered":"2023-10-18T16:07:41.807288+0000","last_clean":"2023-10-18T16:07:41.807288+0000","last_became_active":"2023-10-18T16:07:41.801983+0000","last_became_peered":"2023-10-18T16:07:41.801983+0000","last_unstale":"2023-10-18T16:07:41.807288+0000","last_undegraded":"2023-10-18T16:07:41.807288+0000","last_fullsized":"2023-10-18T16:07:41.807288+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.793283+0000","last_change":"2023-10-18T16:07:36.540468+0000","last_active":"2023-10-18T16:07:41.793283+0000","last_peered":"2023-10-18T16:07:41.793283+0000","last_clean":"2023-10-18T16:07:41.793283+0000","last_became_active":"2023-10-18T16:07:36.540145+0000","last_became_peered":"2023-10-18T16:07:36.540145+0000","last_unstale":"2023-10-18T16:07:41.793283+0000","last_undegraded":"2023-10-18T16:07:41.793283+0000","last_fullsized":"2023-10-18T16:07:41.793283+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:35.522314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:35.522314+0000","last_clean_scrub_stamp":"2023-10-18T16:07:35.522314+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.791238+0000","last_change":"2023-10-18T16:07:41.431117+0000","last_active":"2023-10-18T16:07:41.791238+0000","last_peered":"2023-10-18T16:07:41.791238+0000","last_clean":"2023-10-18T16:07:41.791238+0000","last_became_active":"2023-10-18T16:07:41.430838+0000","last_became_peered":"2023-10-18T16:07:41.430838+0000","last_unstale":"2023-10-18T16:07:41.791238+0000","last_undegraded":"2023-10-18T16:07:41.791238+0000","last_fullsized":"2023-10-18T16:07:41.791238+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.806469+0000","last_change":"2023-10-18T16:07:41.803250+0000","last_active":"2023-10-18T16:07:41.806469+0000","last_peered":"2023-10-18T16:07:41.806469+0000","last_clean":"2023-10-18T16:07:41.806469+0000","last_became_active":"2023-10-18T16:07:41.802930+0000","last_became_peered":"2023-10-18T16:07:41.802930+0000","last_unstale":"2023-10-18T16:07:41.806469+0000","last_undegraded":"2023-10-18T16:07:41.806469+0000","last_fullsized":"2023-10-18T16:07:41.806469+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.807328+0000","last_change":"2023-10-18T16:07:41.803179+0000","last_active":"2023-10-18T16:07:41.807328+0000","last_peered":"2023-10-18T16:07:41.807328+0000","last_clean":"2023-10-18T16:07:41.807328+0000","last_became_active":"2023-10-18T16:07:41.802934+0000","last_became_peered":"2023-10-18T16:07:41.802934+0000","last_unstale":"2023-10-18T16:07:41.807328+0000","last_undegraded":"2023-10-18T16:07:41.807328+0000","last_fullsized":"2023-10-18T16:07:41.807328+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.792656+0000","last_change":"2023-10-18T16:07:41.443157+0000","last_active":"2023-10-18T16:07:41.792656+0000","last_peered":"2023-10-18T16:07:41.792656+0000","last_clean":"2023-10-18T16:07:41.792656+0000","last_became_active":"2023-10-18T16:07:41.442898+0000","last_became_peered":"2023-10-18T16:07:41.442898+0000","last_unstale":"2023-10-18T16:07:41.792656+0000","last_undegraded":"2023-10-18T16:07:41.792656+0000","last_fullsized":"2023-10-18T16:07:41.792656+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.807624+0000","last_change":"2023-10-18T16:07:41.804233+0000","last_active":"2023-10-18T16:07:41.807624+0000","last_peered":"2023-10-18T16:07:41.807624+0000","last_clean":"2023-10-18T16:07:41.807624+0000","last_became_active":"2023-10-18T16:07:41.803966+0000","last_became_peered":"2023-10-18T16:07:41.803966+0000","last_unstale":"2023-10-18T16:07:41.807624+0000","last_undegraded":"2023-10-18T16:07:41.807624+0000","last_fullsized":"2023-10-18T16:07:41.807624+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.790546+0000","last_change":"2023-10-18T16:07:41.443533+0000","last_active":"2023-10-18T16:07:41.790546+0000","last_peered":"2023-10-18T16:07:41.790546+0000","last_clean":"2023-10-18T16:07:41.790546+0000","last_became_active":"2023-10-18T16:07:41.443276+0000","last_became_peered":"2023-10-18T16:07:41.443276+0000","last_unstale":"2023-10-18T16:07:41.790546+0000","last_undegraded":"2023-10-18T16:07:41.790546+0000","last_fullsized":"2023-10-18T16:07:41.790546+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.805621+0000","last_change":"2023-10-18T16:07:41.799449+0000","last_active":"2023-10-18T16:07:41.805621+0000","last_peered":"2023-10-18T16:07:41.805621+0000","last_clean":"2023-10-18T16:07:41.805621+0000","last_became_active":"2023-10-18T16:07:41.799177+0000","last_became_peered":"2023-10-18T16:07:41.799177+0000","last_unstale":"2023-10-18T16:07:41.805621+0000","last_undegraded":"2023-10-18T16:07:41.805621+0000","last_fullsized":"2023-10-18T16:07:41.805621+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.790477+0000","last_change":"2023-10-18T16:07:41.443654+0000","last_active":"2023-10-18T16:07:41.790477+0000","last_peered":"2023-10-18T16:07:41.790477+0000","last_clean":"2023-10-18T16:07:41.790477+0000","last_became_active":"2023-10-18T16:07:41.443453+0000","last_became_peered":"2023-10-18T16:07:41.443453+0000","last_unstale":"2023-10-18T16:07:41.790477+0000","last_undegraded":"2023-10-18T16:07:41.790477+0000","last_fullsized":"2023-10-18T16:07:41.790477+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.802526+0000","last_change":"2023-10-18T16:07:41.798748+0000","last_active":"2023-10-18T16:07:41.802526+0000","last_peered":"2023-10-18T16:07:41.802526+0000","last_clean":"2023-10-18T16:07:41.802526+0000","last_became_active":"2023-10-18T16:07:41.798479+0000","last_became_peered":"2023-10-18T16:07:41.798479+0000","last_unstale":"2023-10-18T16:07:41.802526+0000","last_undegraded":"2023-10-18T16:07:41.802526+0000","last_fullsized":"2023-10-18T16:07:41.802526+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.788543+0000","last_change":"2023-10-18T16:07:41.438072+0000","last_active":"2023-10-18T16:07:41.788543+0000","last_peered":"2023-10-18T16:07:41.788543+0000","last_clean":"2023-10-18T16:07:41.788543+0000","last_became_active":"2023-10-18T16:07:41.437665+0000","last_became_peered":"2023-10-18T16:07:41.437665+0000","last_unstale":"2023-10-18T16:07:41.788543+0000","last_undegraded":"2023-10-18T16:07:41.788543+0000","last_fullsized":"2023-10-18T16:07:41.788543+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.806103+0000","last_change":"2023-10-18T16:07:41.800789+0000","last_active":"2023-10-18T16:07:41.806103+0000","last_peered":"2023-10-18T16:07:41.806103+0000","last_clean":"2023-10-18T16:07:41.806103+0000","last_became_active":"2023-10-18T16:07:41.800336+0000","last_became_peered":"2023-10-18T16:07:41.800336+0000","last_unstale":"2023-10-18T16:07:41.806103+0000","last_undegraded":"2023-10-18T16:07:41.806103+0000","last_fullsized":"2023-10-18T16:07:41.806103+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.790623+0000","last_change":"2023-10-18T16:07:41.439886+0000","last_active":"2023-10-18T16:07:41.790623+0000","last_peered":"2023-10-18T16:07:41.790623+0000","last_clean":"2023-10-18T16:07:41.790623+0000","last_became_active":"2023-10-18T16:07:41.439664+0000","last_became_peered":"2023-10-18T16:07:41.439664+0000","last_unstale":"2023-10-18T16:07:41.790623+0000","last_undegraded":"2023-10-18T16:07:41.790623+0000","last_fullsized":"2023-10-18T16:07:41.790623+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.803656+0000","last_change":"2023-10-18T16:07:41.797792+0000","last_active":"2023-10-18T16:07:41.803656+0000","last_peered":"2023-10-18T16:07:41.803656+0000","last_clean":"2023-10-18T16:07:41.803656+0000","last_became_active":"2023-10-18T16:07:41.797498+0000","last_became_peered":"2023-10-18T16:07:41.797498+0000","last_unstale":"2023-10-18T16:07:41.803656+0000","last_undegraded":"2023-10-18T16:07:41.803656+0000","last_fullsized":"2023-10-18T16:07:41.803656+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.793059+0000","last_change":"2023-10-18T16:07:41.434136+0000","last_active":"2023-10-18T16:07:41.793059+0000","last_peered":"2023-10-18T16:07:41.793059+0000","last_clean":"2023-10-18T16:07:41.793059+0000","last_became_active":"2023-10-18T16:07:41.433862+0000","last_became_peered":"2023-10-18T16:07:41.433862+0000","last_unstale":"2023-10-18T16:07:41.793059+0000","last_undegraded":"2023-10-18T16:07:41.793059+0000","last_fullsized":"2023-10-18T16:07:41.793059+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.801349+0000","last_change":"2023-10-18T16:07:41.797301+0000","last_active":"2023-10-18T16:07:41.801349+0000","last_peered":"2023-10-18T16:07:41.801349+0000","last_clean":"2023-10-18T16:07:41.801349+0000","last_became_active":"2023-10-18T16:07:41.797029+0000","last_became_peered":"2023-10-18T16:07:41.797029+0000","last_unstale":"2023-10-18T16:07:41.801349+0000","last_undegraded":"2023-10-18T16:07:41.801349+0000","last_fullsized":"2023-10-18T16:07:41.801349+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.795552+0000","last_change":"2023-10-18T16:07:41.426435+0000","last_active":"2023-10-18T16:07:41.795552+0000","last_peered":"2023-10-18T16:07:41.795552+0000","last_clean":"2023-10-18T16:07:41.795552+0000","last_became_active":"2023-10-18T16:07:41.426169+0000","last_became_peered":"2023-10-18T16:07:41.426169+0000","last_unstale":"2023-10-18T16:07:41.795552+0000","last_undegraded":"2023-10-18T16:07:41.795552+0000","last_fullsized":"2023-10-18T16:07:41.795552+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.809706+0000","last_change":"2023-10-18T16:07:41.808005+0000","last_active":"2023-10-18T16:07:41.809706+0000","last_peered":"2023-10-18T16:07:41.809706+0000","last_clean":"2023-10-18T16:07:41.809706+0000","last_became_active":"2023-10-18T16:07:41.807155+0000","last_became_peered":"2023-10-18T16:07:41.807155+0000","last_unstale":"2023-10-18T16:07:41.809706+0000","last_undegraded":"2023-10-18T16:07:41.809706+0000","last_fullsized":"2023-10-18T16:07:41.809706+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.790085+0000","last_change":"2023-10-18T16:07:41.437945+0000","last_active":"2023-10-18T16:07:41.790085+0000","last_peered":"2023-10-18T16:07:41.790085+0000","last_clean":"2023-10-18T16:07:41.790085+0000","last_became_active":"2023-10-18T16:07:41.437118+0000","last_became_peered":"2023-10-18T16:07:41.437118+0000","last_unstale":"2023-10-18T16:07:41.790085+0000","last_undegraded":"2023-10-18T16:07:41.790085+0000","last_fullsized":"2023-10-18T16:07:41.790085+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.807377+0000","last_change":"2023-10-18T16:07:41.801600+0000","last_active":"2023-10-18T16:07:41.807377+0000","last_peered":"2023-10-18T16:07:41.807377+0000","last_clean":"2023-10-18T16:07:41.807377+0000","last_became_active":"2023-10-18T16:07:41.801002+0000","last_became_peered":"2023-10-18T16:07:41.801002+0000","last_unstale":"2023-10-18T16:07:41.807377+0000","last_undegraded":"2023-10-18T16:07:41.807377+0000","last_fullsized":"2023-10-18T16:07:41.807377+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.792471+0000","last_change":"2023-10-18T16:07:41.432368+0000","last_active":"2023-10-18T16:07:41.792471+0000","last_peered":"2023-10-18T16:07:41.792471+0000","last_clean":"2023-10-18T16:07:41.792471+0000","last_became_active":"2023-10-18T16:07:41.431923+0000","last_became_peered":"2023-10-18T16:07:41.431923+0000","last_unstale":"2023-10-18T16:07:41.792471+0000","last_undegraded":"2023-10-18T16:07:41.792471+0000","last_fullsized":"2023-10-18T16:07:41.792471+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.810702+0000","last_change":"2023-10-18T16:07:41.809593+0000","last_active":"2023-10-18T16:07:41.810702+0000","last_peered":"2023-10-18T16:07:41.810702+0000","last_clean":"2023-10-18T16:07:41.810702+0000","last_became_active":"2023-10-18T16:07:41.809309+0000","last_became_peered":"2023-10-18T16:07:41.809309+0000","last_unstale":"2023-10-18T16:07:41.810702+0000","last_undegraded":"2023-10-18T16:07:41.810702+0000","last_fullsized":"2023-10-18T16:07:41.810702+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.789348+0000","last_change":"2023-10-18T16:07:41.443916+0000","last_active":"2023-10-18T16:07:41.789348+0000","last_peered":"2023-10-18T16:07:41.789348+0000","last_clean":"2023-10-18T16:07:41.789348+0000","last_became_active":"2023-10-18T16:07:41.443695+0000","last_became_peered":"2023-10-18T16:07:41.443695+0000","last_unstale":"2023-10-18T16:07:41.789348+0000","last_undegraded":"2023-10-18T16:07:41.789348+0000","last_fullsized":"2023-10-18T16:07:41.789348+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.806255+0000","last_change":"2023-10-18T16:07:41.803821+0000","last_active":"2023-10-18T16:07:41.806255+0000","last_peered":"2023-10-18T16:07:41.806255+0000","last_clean":"2023-10-18T16:07:41.806255+0000","last_became_active":"2023-10-18T16:07:41.803540+0000","last_became_peered":"2023-10-18T16:07:41.803540+0000","last_unstale":"2023-10-18T16:07:41.806255+0000","last_undegraded":"2023-10-18T16:07:41.806255+0000","last_fullsized":"2023-10-18T16:07:41.806255+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.786238+0000","last_change":"2023-10-18T16:07:41.429591+0000","last_active":"2023-10-18T16:07:41.786238+0000","last_peered":"2023-10-18T16:07:41.786238+0000","last_clean":"2023-10-18T16:07:41.786238+0000","last_became_active":"2023-10-18T16:07:41.429290+0000","last_became_peered":"2023-10-18T16:07:41.429290+0000","last_unstale":"2023-10-18T16:07:41.786238+0000","last_undegraded":"2023-10-18T16:07:41.786238+0000","last_fullsized":"2023-10-18T16:07:41.786238+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.799825+0000","last_change":"2023-10-18T16:07:41.795679+0000","last_active":"2023-10-18T16:07:41.799825+0000","last_peered":"2023-10-18T16:07:41.799825+0000","last_clean":"2023-10-18T16:07:41.799825+0000","last_became_active":"2023-10-18T16:07:41.795413+0000","last_became_peered":"2023-10-18T16:07:41.795413+0000","last_unstale":"2023-10-18T16:07:41.799825+0000","last_undegraded":"2023-10-18T16:07:41.799825+0000","last_fullsized":"2023-10-18T16:07:41.799825+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.793196+0000","last_change":"2023-10-18T16:07:41.436812+0000","last_active":"2023-10-18T16:07:41.793196+0000","last_peered":"2023-10-18T16:07:41.793196+0000","last_clean":"2023-10-18T16:07:41.793196+0000","last_became_active":"2023-10-18T16:07:41.436541+0000","last_became_peered":"2023-10-18T16:07:41.436541+0000","last_unstale":"2023-10-18T16:07:41.793196+0000","last_undegraded":"2023-10-18T16:07:41.793196+0000","last_fullsized":"2023-10-18T16:07:41.793196+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.806410+0000","last_change":"2023-10-18T16:07:41.803115+0000","last_active":"2023-10-18T16:07:41.806410+0000","last_peered":"2023-10-18T16:07:41.806410+0000","last_clean":"2023-10-18T16:07:41.806410+0000","last_became_active":"2023-10-18T16:07:41.802828+0000","last_became_peered":"2023-10-18T16:07:41.802828+0000","last_unstale":"2023-10-18T16:07:41.806410+0000","last_undegraded":"2023-10-18T16:07:41.806410+0000","last_fullsized":"2023-10-18T16:07:41.806410+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.789662+0000","last_change":"2023-10-18T16:07:41.443476+0000","last_active":"2023-10-18T16:07:41.789662+0000","last_peered":"2023-10-18T16:07:41.789662+0000","last_clean":"2023-10-18T16:07:41.789662+0000","last_became_active":"2023-10-18T16:07:41.443295+0000","last_became_peered":"2023-10-18T16:07:41.443295+0000","last_unstale":"2023-10-18T16:07:41.789662+0000","last_undegraded":"2023-10-18T16:07:41.789662+0000","last_fullsized":"2023-10-18T16:07:41.789662+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.805189+0000","last_change":"2023-10-18T16:07:41.800058+0000","last_active":"2023-10-18T16:07:41.805189+0000","last_peered":"2023-10-18T16:07:41.805189+0000","last_clean":"2023-10-18T16:07:41.805189+0000","last_became_active":"2023-10-18T16:07:41.799452+0000","last_became_peered":"2023-10-18T16:07:41.799452+0000","last_unstale":"2023-10-18T16:07:41.805189+0000","last_undegraded":"2023-10-18T16:07:41.805189+0000","last_fullsized":"2023-10-18T16:07:41.805189+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.789303+0000","last_change":"2023-10-18T16:07:41.440724+0000","last_active":"2023-10-18T16:07:41.789303+0000","last_peered":"2023-10-18T16:07:41.789303+0000","last_clean":"2023-10-18T16:07:41.789303+0000","last_became_active":"2023-10-18T16:07:41.440435+0000","last_became_peered":"2023-10-18T16:07:41.440435+0000","last_unstale":"2023-10-18T16:07:41.789303+0000","last_undegraded":"2023-10-18T16:07:41.789303+0000","last_fullsized":"2023-10-18T16:07:41.789303+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.806559+0000","last_change":"2023-10-18T16:07:41.804438+0000","last_active":"2023-10-18T16:07:41.806559+0000","last_peered":"2023-10-18T16:07:41.806559+0000","last_clean":"2023-10-18T16:07:41.806559+0000","last_became_active":"2023-10-18T16:07:41.804126+0000","last_became_peered":"2023-10-18T16:07:41.804126+0000","last_unstale":"2023-10-18T16:07:41.806559+0000","last_undegraded":"2023-10-18T16:07:41.806559+0000","last_fullsized":"2023-10-18T16:07:41.806559+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.788864+0000","last_change":"2023-10-18T16:07:41.447812+0000","last_active":"2023-10-18T16:07:41.788864+0000","last_peered":"2023-10-18T16:07:41.788864+0000","last_clean":"2023-10-18T16:07:41.788864+0000","last_became_active":"2023-10-18T16:07:41.447479+0000","last_became_peered":"2023-10-18T16:07:41.447479+0000","last_unstale":"2023-10-18T16:07:41.788864+0000","last_undegraded":"2023-10-18T16:07:41.788864+0000","last_fullsized":"2023-10-18T16:07:41.788864+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.804861+0000","last_change":"2023-10-18T16:07:41.801084+0000","last_active":"2023-10-18T16:07:41.804861+0000","last_peered":"2023-10-18T16:07:41.804861+0000","last_clean":"2023-10-18T16:07:41.804861+0000","last_became_active":"2023-10-18T16:07:41.800783+0000","last_became_peered":"2023-10-18T16:07:41.800783+0000","last_unstale":"2023-10-18T16:07:41.804861+0000","last_undegraded":"2023-10-18T16:07:41.804861+0000","last_fullsized":"2023-10-18T16:07:41.804861+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.790315+0000","last_change":"2023-10-18T16:07:41.437755+0000","last_active":"2023-10-18T16:07:41.790315+0000","last_peered":"2023-10-18T16:07:41.790315+0000","last_clean":"2023-10-18T16:07:41.790315+0000","last_became_active":"2023-10-18T16:07:41.436781+0000","last_became_peered":"2023-10-18T16:07:41.436781+0000","last_unstale":"2023-10-18T16:07:41.790315+0000","last_undegraded":"2023-10-18T16:07:41.790315+0000","last_fullsized":"2023-10-18T16:07:41.790315+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.810938+0000","last_change":"2023-10-18T16:07:41.808318+0000","last_active":"2023-10-18T16:07:41.810938+0000","last_peered":"2023-10-18T16:07:41.810938+0000","last_clean":"2023-10-18T16:07:41.810938+0000","last_became_active":"2023-10-18T16:07:41.807983+0000","last_became_peered":"2023-10-18T16:07:41.807983+0000","last_unstale":"2023-10-18T16:07:41.810938+0000","last_undegraded":"2023-10-18T16:07:41.810938+0000","last_fullsized":"2023-10-18T16:07:41.810938+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.792563+0000","last_change":"2023-10-18T16:07:41.442119+0000","last_active":"2023-10-18T16:07:41.792563+0000","last_peered":"2023-10-18T16:07:41.792563+0000","last_clean":"2023-10-18T16:07:41.792563+0000","last_became_active":"2023-10-18T16:07:41.441816+0000","last_became_peered":"2023-10-18T16:07:41.441816+0000","last_unstale":"2023-10-18T16:07:41.792563+0000","last_undegraded":"2023-10-18T16:07:41.792563+0000","last_fullsized":"2023-10-18T16:07:41.792563+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.810523+0000","last_change":"2023-10-18T16:07:41.809378+0000","last_active":"2023-10-18T16:07:41.810523+0000","last_peered":"2023-10-18T16:07:41.810523+0000","last_clean":"2023-10-18T16:07:41.810523+0000","last_became_active":"2023-10-18T16:07:41.809102+0000","last_became_peered":"2023-10-18T16:07:41.809102+0000","last_unstale":"2023-10-18T16:07:41.810523+0000","last_undegraded":"2023-10-18T16:07:41.810523+0000","last_fullsized":"2023-10-18T16:07:41.810523+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.793639+0000","last_change":"2023-10-18T16:07:41.442792+0000","last_active":"2023-10-18T16:07:41.793639+0000","last_peered":"2023-10-18T16:07:41.793639+0000","last_clean":"2023-10-18T16:07:41.793639+0000","last_became_active":"2023-10-18T16:07:41.442506+0000","last_became_peered":"2023-10-18T16:07:41.442506+0000","last_unstale":"2023-10-18T16:07:41.793639+0000","last_undegraded":"2023-10-18T16:07:41.793639+0000","last_fullsized":"2023-10-18T16:07:41.793639+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.807999+0000","last_change":"2023-10-18T16:07:41.804704+0000","last_active":"2023-10-18T16:07:41.807999+0000","last_peered":"2023-10-18T16:07:41.807999+0000","last_clean":"2023-10-18T16:07:41.807999+0000","last_became_active":"2023-10-18T16:07:41.804437+0000","last_became_peered":"2023-10-18T16:07:41.804437+0000","last_unstale":"2023-10-18T16:07:41.807999+0000","last_undegraded":"2023-10-18T16:07:41.807999+0000","last_fullsized":"2023-10-18T16:07:41.807999+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.798211+0000","last_change":"2023-10-18T16:07:41.436171+0000","last_active":"2023-10-18T16:07:41.798211+0000","last_peered":"2023-10-18T16:07:41.798211+0000","last_clean":"2023-10-18T16:07:41.798211+0000","last_became_active":"2023-10-18T16:07:41.435844+0000","last_became_peered":"2023-10-18T16:07:41.435844+0000","last_unstale":"2023-10-18T16:07:41.798211+0000","last_undegraded":"2023-10-18T16:07:41.798211+0000","last_fullsized":"2023-10-18T16:07:41.798211+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.802055+0000","last_change":"2023-10-18T16:07:41.796556+0000","last_active":"2023-10-18T16:07:41.802055+0000","last_peered":"2023-10-18T16:07:41.802055+0000","last_clean":"2023-10-18T16:07:41.802055+0000","last_became_active":"2023-10-18T16:07:41.796291+0000","last_became_peered":"2023-10-18T16:07:41.796291+0000","last_unstale":"2023-10-18T16:07:41.802055+0000","last_undegraded":"2023-10-18T16:07:41.802055+0000","last_fullsized":"2023-10-18T16:07:41.802055+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.795432+0000","last_change":"2023-10-18T16:07:41.434332+0000","last_active":"2023-10-18T16:07:41.795432+0000","last_peered":"2023-10-18T16:07:41.795432+0000","last_clean":"2023-10-18T16:07:41.795432+0000","last_became_active":"2023-10-18T16:07:41.434076+0000","last_became_peered":"2023-10-18T16:07:41.434076+0000","last_unstale":"2023-10-18T16:07:41.795432+0000","last_undegraded":"2023-10-18T16:07:41.795432+0000","last_fullsized":"2023-10-18T16:07:41.795432+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.808487+0000","last_change":"2023-10-18T16:07:41.806379+0000","last_active":"2023-10-18T16:07:41.808487+0000","last_peered":"2023-10-18T16:07:41.808487+0000","last_clean":"2023-10-18T16:07:41.808487+0000","last_became_active":"2023-10-18T16:07:41.806018+0000","last_became_peered":"2023-10-18T16:07:41.806018+0000","last_unstale":"2023-10-18T16:07:41.808487+0000","last_undegraded":"2023-10-18T16:07:41.808487+0000","last_fullsized":"2023-10-18T16:07:41.808487+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.790761+0000","last_change":"2023-10-18T16:07:41.438688+0000","last_active":"2023-10-18T16:07:41.790761+0000","last_peered":"2023-10-18T16:07:41.790761+0000","last_clean":"2023-10-18T16:07:41.790761+0000","last_became_active":"2023-10-18T16:07:41.438405+0000","last_became_peered":"2023-10-18T16:07:41.438405+0000","last_unstale":"2023-10-18T16:07:41.790761+0000","last_undegraded":"2023-10-18T16:07:41.790761+0000","last_fullsized":"2023-10-18T16:07:41.790761+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.805209+0000","last_change":"2023-10-18T16:07:41.800230+0000","last_active":"2023-10-18T16:07:41.805209+0000","last_peered":"2023-10-18T16:07:41.805209+0000","last_clean":"2023-10-18T16:07:41.805209+0000","last_became_active":"2023-10-18T16:07:41.799787+0000","last_became_peered":"2023-10-18T16:07:41.799787+0000","last_unstale":"2023-10-18T16:07:41.805209+0000","last_undegraded":"2023-10-18T16:07:41.805209+0000","last_fullsized":"2023-10-18T16:07:41.805209+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.789370+0000","last_change":"2023-10-18T16:07:41.440794+0000","last_active":"2023-10-18T16:07:41.789370+0000","last_peered":"2023-10-18T16:07:41.789370+0000","last_clean":"2023-10-18T16:07:41.789370+0000","last_became_active":"2023-10-18T16:07:41.440412+0000","last_became_peered":"2023-10-18T16:07:41.440412+0000","last_unstale":"2023-10-18T16:07:41.789370+0000","last_undegraded":"2023-10-18T16:07:41.789370+0000","last_fullsized":"2023-10-18T16:07:41.789370+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.806876+0000","last_change":"2023-10-18T16:07:41.801402+0000","last_active":"2023-10-18T16:07:41.806876+0000","last_peered":"2023-10-18T16:07:41.806876+0000","last_clean":"2023-10-18T16:07:41.806876+0000","last_became_active":"2023-10-18T16:07:41.800562+0000","last_became_peered":"2023-10-18T16:07:41.800562+0000","last_unstale":"2023-10-18T16:07:41.806876+0000","last_undegraded":"2023-10-18T16:07:41.806876+0000","last_fullsized":"2023-10-18T16:07:41.806876+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.785692+0000","last_change":"2023-10-18T16:07:41.445361+0000","last_active":"2023-10-18T16:07:41.785692+0000","last_peered":"2023-10-18T16:07:41.785692+0000","last_clean":"2023-10-18T16:07:41.785692+0000","last_became_active":"2023-10-18T16:07:41.445159+0000","last_became_peered":"2023-10-18T16:07:41.445159+0000","last_unstale":"2023-10-18T16:07:41.785692+0000","last_undegraded":"2023-10-18T16:07:41.785692+0000","last_fullsized":"2023-10-18T16:07:41.785692+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.806606+0000","last_change":"2023-10-18T16:07:41.806046+0000","last_active":"2023-10-18T16:07:41.806606+0000","last_peered":"2023-10-18T16:07:41.806606+0000","last_clean":"2023-10-18T16:07:41.806606+0000","last_became_active":"2023-10-18T16:07:41.805799+0000","last_became_peered":"2023-10-18T16:07:41.805799+0000","last_unstale":"2023-10-18T16:07:41.806606+0000","last_undegraded":"2023-10-18T16:07:41.806606+0000","last_fullsized":"2023-10-18T16:07:41.806606+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.791286+0000","last_change":"2023-10-18T16:07:41.425946+0000","last_active":"2023-10-18T16:07:41.791286+0000","last_peered":"2023-10-18T16:07:41.791286+0000","last_clean":"2023-10-18T16:07:41.791286+0000","last_became_active":"2023-10-18T16:07:41.425673+0000","last_became_peered":"2023-10-18T16:07:41.425673+0000","last_unstale":"2023-10-18T16:07:41.791286+0000","last_undegraded":"2023-10-18T16:07:41.791286+0000","last_fullsized":"2023-10-18T16:07:41.791286+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.811135+0000","last_change":"2023-10-18T16:07:41.810501+0000","last_active":"2023-10-18T16:07:41.811135+0000","last_peered":"2023-10-18T16:07:41.811135+0000","last_clean":"2023-10-18T16:07:41.811135+0000","last_became_active":"2023-10-18T16:07:41.810233+0000","last_became_peered":"2023-10-18T16:07:41.810233+0000","last_unstale":"2023-10-18T16:07:41.811135+0000","last_undegraded":"2023-10-18T16:07:41.811135+0000","last_fullsized":"2023-10-18T16:07:41.811135+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.788717+0000","last_change":"2023-10-18T16:07:41.434399+0000","last_active":"2023-10-18T16:07:41.788717+0000","last_peered":"2023-10-18T16:07:41.788717+0000","last_clean":"2023-10-18T16:07:41.788717+0000","last_became_active":"2023-10-18T16:07:41.434143+0000","last_became_peered":"2023-10-18T16:07:41.434143+0000","last_unstale":"2023-10-18T16:07:41.788717+0000","last_undegraded":"2023-10-18T16:07:41.788717+0000","last_fullsized":"2023-10-18T16:07:41.788717+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.801283+0000","last_change":"2023-10-18T16:07:41.799082+0000","last_active":"2023-10-18T16:07:41.801283+0000","last_peered":"2023-10-18T16:07:41.801283+0000","last_clean":"2023-10-18T16:07:41.801283+0000","last_became_active":"2023-10-18T16:07:41.798807+0000","last_became_peered":"2023-10-18T16:07:41.798807+0000","last_unstale":"2023-10-18T16:07:41.801283+0000","last_undegraded":"2023-10-18T16:07:41.801283+0000","last_fullsized":"2023-10-18T16:07:41.801283+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.796407+0000","last_change":"2023-10-18T16:07:41.445579+0000","last_active":"2023-10-18T16:07:41.796407+0000","last_peered":"2023-10-18T16:07:41.796407+0000","last_clean":"2023-10-18T16:07:41.796407+0000","last_became_active":"2023-10-18T16:07:41.445348+0000","last_became_peered":"2023-10-18T16:07:41.445348+0000","last_unstale":"2023-10-18T16:07:41.796407+0000","last_undegraded":"2023-10-18T16:07:41.796407+0000","last_fullsized":"2023-10-18T16:07:41.796407+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.807820+0000","last_change":"2023-10-18T16:07:41.802347+0000","last_active":"2023-10-18T16:07:41.807820+0000","last_peered":"2023-10-18T16:07:41.807820+0000","last_clean":"2023-10-18T16:07:41.807820+0000","last_became_active":"2023-10-18T16:07:41.802061+0000","last_became_peered":"2023-10-18T16:07:41.802061+0000","last_unstale":"2023-10-18T16:07:41.807820+0000","last_undegraded":"2023-10-18T16:07:41.807820+0000","last_fullsized":"2023-10-18T16:07:41.807820+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.793332+0000","last_change":"2023-10-18T16:07:41.439414+0000","last_active":"2023-10-18T16:07:41.793332+0000","last_peered":"2023-10-18T16:07:41.793332+0000","last_clean":"2023-10-18T16:07:41.793332+0000","last_became_active":"2023-10-18T16:07:41.439137+0000","last_became_peered":"2023-10-18T16:07:41.439137+0000","last_unstale":"2023-10-18T16:07:41.793332+0000","last_undegraded":"2023-10-18T16:07:41.793332+0000","last_fullsized":"2023-10-18T16:07:41.793332+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.810429+0000","last_change":"2023-10-18T16:07:41.808426+0000","last_active":"2023-10-18T16:07:41.810429+0000","last_peered":"2023-10-18T16:07:41.810429+0000","last_clean":"2023-10-18T16:07:41.810429+0000","last_became_active":"2023-10-18T16:07:41.808146+0000","last_became_peered":"2023-10-18T16:07:41.808146+0000","last_unstale":"2023-10-18T16:07:41.810429+0000","last_undegraded":"2023-10-18T16:07:41.810429+0000","last_fullsized":"2023-10-18T16:07:41.810429+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.793112+0000","last_change":"2023-10-18T16:07:41.439810+0000","last_active":"2023-10-18T16:07:41.793112+0000","last_peered":"2023-10-18T16:07:41.793112+0000","last_clean":"2023-10-18T16:07:41.793112+0000","last_became_active":"2023-10-18T16:07:41.439545+0000","last_became_peered":"2023-10-18T16:07:41.439545+0000","last_unstale":"2023-10-18T16:07:41.793112+0000","last_undegraded":"2023-10-18T16:07:41.793112+0000","last_fullsized":"2023-10-18T16:07:41.793112+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.806045+0000","last_change":"2023-10-18T16:07:41.800617+0000","last_active":"2023-10-18T16:07:41.806045+0000","last_peered":"2023-10-18T16:07:41.806045+0000","last_clean":"2023-10-18T16:07:41.806045+0000","last_became_active":"2023-10-18T16:07:41.799998+0000","last_became_peered":"2023-10-18T16:07:41.799998+0000","last_unstale":"2023-10-18T16:07:41.806045+0000","last_undegraded":"2023-10-18T16:07:41.806045+0000","last_fullsized":"2023-10-18T16:07:41.806045+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.798396+0000","last_change":"2023-10-18T16:07:41.442796+0000","last_active":"2023-10-18T16:07:41.798396+0000","last_peered":"2023-10-18T16:07:41.798396+0000","last_clean":"2023-10-18T16:07:41.798396+0000","last_became_active":"2023-10-18T16:07:41.442007+0000","last_became_peered":"2023-10-18T16:07:41.442007+0000","last_unstale":"2023-10-18T16:07:41.798396+0000","last_undegraded":"2023-10-18T16:07:41.798396+0000","last_fullsized":"2023-10-18T16:07:41.798396+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3f","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.801310+0000","last_change":"2023-10-18T16:07:41.799046+0000","last_active":"2023-10-18T16:07:41.801310+0000","last_peered":"2023-10-18T16:07:41.801310+0000","last_clean":"2023-10-18T16:07:41.801310+0000","last_became_active":"2023-10-18T16:07:41.798754+0000","last_became_peered":"2023-10-18T16:07:41.798754+0000","last_unstale":"2023-10-18T16:07:41.801310+0000","last_undegraded":"2023-10-18T16:07:41.801310+0000","last_fullsized":"2023-10-18T16:07:41.801310+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.789919+0000","last_change":"2023-10-18T16:07:41.441443+0000","last_active":"2023-10-18T16:07:41.789919+0000","last_peered":"2023-10-18T16:07:41.789919+0000","last_clean":"2023-10-18T16:07:41.789919+0000","last_became_active":"2023-10-18T16:07:41.441208+0000","last_became_peered":"2023-10-18T16:07:41.441208+0000","last_unstale":"2023-10-18T16:07:41.789919+0000","last_undegraded":"2023-10-18T16:07:41.789919+0000","last_fullsized":"2023-10-18T16:07:41.789919+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.806457+0000","last_change":"2023-10-18T16:07:41.800460+0000","last_active":"2023-10-18T16:07:41.806457+0000","last_peered":"2023-10-18T16:07:41.806457+0000","last_clean":"2023-10-18T16:07:41.806457+0000","last_became_active":"2023-10-18T16:07:41.800013+0000","last_became_peered":"2023-10-18T16:07:41.800013+0000","last_unstale":"2023-10-18T16:07:41.806457+0000","last_undegraded":"2023-10-18T16:07:41.806457+0000","last_fullsized":"2023-10-18T16:07:41.806457+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.799786+0000","last_change":"2023-10-18T16:07:41.438711+0000","last_active":"2023-10-18T16:07:41.799786+0000","last_peered":"2023-10-18T16:07:41.799786+0000","last_clean":"2023-10-18T16:07:41.799786+0000","last_became_active":"2023-10-18T16:07:41.438408+0000","last_became_peered":"2023-10-18T16:07:41.438408+0000","last_unstale":"2023-10-18T16:07:41.799786+0000","last_undegraded":"2023-10-18T16:07:41.799786+0000","last_fullsized":"2023-10-18T16:07:41.799786+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.806161+0000","last_change":"2023-10-18T16:07:41.800718+0000","last_active":"2023-10-18T16:07:41.806161+0000","last_peered":"2023-10-18T16:07:41.806161+0000","last_clean":"2023-10-18T16:07:41.806161+0000","last_became_active":"2023-10-18T16:07:41.800400+0000","last_became_peered":"2023-10-18T16:07:41.800400+0000","last_unstale":"2023-10-18T16:07:41.806161+0000","last_undegraded":"2023-10-18T16:07:41.806161+0000","last_fullsized":"2023-10-18T16:07:41.806161+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.790414+0000","last_change":"2023-10-18T16:07:41.438151+0000","last_active":"2023-10-18T16:07:41.790414+0000","last_peered":"2023-10-18T16:07:41.790414+0000","last_clean":"2023-10-18T16:07:41.790414+0000","last_became_active":"2023-10-18T16:07:41.437504+0000","last_became_peered":"2023-10-18T16:07:41.437504+0000","last_unstale":"2023-10-18T16:07:41.790414+0000","last_undegraded":"2023-10-18T16:07:41.790414+0000","last_fullsized":"2023-10-18T16:07:41.790414+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.805013+0000","last_change":"2023-10-18T16:07:41.802411+0000","last_active":"2023-10-18T16:07:41.805013+0000","last_peered":"2023-10-18T16:07:41.805013+0000","last_clean":"2023-10-18T16:07:41.805013+0000","last_became_active":"2023-10-18T16:07:41.802124+0000","last_became_peered":"2023-10-18T16:07:41.802124+0000","last_unstale":"2023-10-18T16:07:41.805013+0000","last_undegraded":"2023-10-18T16:07:41.805013+0000","last_fullsized":"2023-10-18T16:07:41.805013+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.797104+0000","last_change":"2023-10-18T16:07:41.439071+0000","last_active":"2023-10-18T16:07:41.797104+0000","last_peered":"2023-10-18T16:07:41.797104+0000","last_clean":"2023-10-18T16:07:41.797104+0000","last_became_active":"2023-10-18T16:07:41.438144+0000","last_became_peered":"2023-10-18T16:07:41.438144+0000","last_unstale":"2023-10-18T16:07:41.797104+0000","last_undegraded":"2023-10-18T16:07:41.797104+0000","last_fullsized":"2023-10-18T16:07:41.797104+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.804818+0000","last_change":"2023-10-18T16:07:41.798403+0000","last_active":"2023-10-18T16:07:41.804818+0000","last_peered":"2023-10-18T16:07:41.804818+0000","last_clean":"2023-10-18T16:07:41.804818+0000","last_became_active":"2023-10-18T16:07:41.798149+0000","last_became_peered":"2023-10-18T16:07:41.798149+0000","last_unstale":"2023-10-18T16:07:41.804818+0000","last_undegraded":"2023-10-18T16:07:41.804818+0000","last_fullsized":"2023-10-18T16:07:41.804818+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.790253+0000","last_change":"2023-10-18T16:07:41.432914+0000","last_active":"2023-10-18T16:07:41.790253+0000","last_peered":"2023-10-18T16:07:41.790253+0000","last_clean":"2023-10-18T16:07:41.790253+0000","last_became_active":"2023-10-18T16:07:41.432601+0000","last_became_peered":"2023-10-18T16:07:41.432601+0000","last_unstale":"2023-10-18T16:07:41.790253+0000","last_undegraded":"2023-10-18T16:07:41.790253+0000","last_fullsized":"2023-10-18T16:07:41.790253+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.806049+0000","last_change":"2023-10-18T16:07:41.801443+0000","last_active":"2023-10-18T16:07:41.806049+0000","last_peered":"2023-10-18T16:07:41.806049+0000","last_clean":"2023-10-18T16:07:41.806049+0000","last_became_active":"2023-10-18T16:07:41.801271+0000","last_became_peered":"2023-10-18T16:07:41.801271+0000","last_unstale":"2023-10-18T16:07:41.806049+0000","last_undegraded":"2023-10-18T16:07:41.806049+0000","last_fullsized":"2023-10-18T16:07:41.806049+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.790644+0000","last_change":"2023-10-18T16:07:41.434525+0000","last_active":"2023-10-18T16:07:41.790644+0000","last_peered":"2023-10-18T16:07:41.790644+0000","last_clean":"2023-10-18T16:07:41.790644+0000","last_became_active":"2023-10-18T16:07:41.434220+0000","last_became_peered":"2023-10-18T16:07:41.434220+0000","last_unstale":"2023-10-18T16:07:41.790644+0000","last_undegraded":"2023-10-18T16:07:41.790644+0000","last_fullsized":"2023-10-18T16:07:41.790644+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.808917+0000","last_change":"2023-10-18T16:07:41.807740+0000","last_active":"2023-10-18T16:07:41.808917+0000","last_peered":"2023-10-18T16:07:41.808917+0000","last_clean":"2023-10-18T16:07:41.808917+0000","last_became_active":"2023-10-18T16:07:41.807473+0000","last_became_peered":"2023-10-18T16:07:41.807473+0000","last_unstale":"2023-10-18T16:07:41.808917+0000","last_undegraded":"2023-10-18T16:07:41.808917+0000","last_fullsized":"2023-10-18T16:07:41.808917+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.785962+0000","last_change":"2023-10-18T16:07:41.427326+0000","last_active":"2023-10-18T16:07:41.785962+0000","last_peered":"2023-10-18T16:07:41.785962+0000","last_clean":"2023-10-18T16:07:41.785962+0000","last_became_active":"2023-10-18T16:07:41.427083+0000","last_became_peered":"2023-10-18T16:07:41.427083+0000","last_unstale":"2023-10-18T16:07:41.785962+0000","last_undegraded":"2023-10-18T16:07:41.785962+0000","last_fullsized":"2023-10-18T16:07:41.785962+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.807852+0000","last_change":"2023-10-18T16:07:41.804453+0000","last_active":"2023-10-18T16:07:41.807852+0000","last_peered":"2023-10-18T16:07:41.807852+0000","last_clean":"2023-10-18T16:07:41.807852+0000","last_became_active":"2023-10-18T16:07:41.802577+0000","last_became_peered":"2023-10-18T16:07:41.802577+0000","last_unstale":"2023-10-18T16:07:41.807852+0000","last_undegraded":"2023-10-18T16:07:41.807852+0000","last_fullsized":"2023-10-18T16:07:41.807852+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.798474+0000","last_change":"2023-10-18T16:07:41.436757+0000","last_active":"2023-10-18T16:07:41.798474+0000","last_peered":"2023-10-18T16:07:41.798474+0000","last_clean":"2023-10-18T16:07:41.798474+0000","last_became_active":"2023-10-18T16:07:41.436423+0000","last_became_peered":"2023-10-18T16:07:41.436423+0000","last_unstale":"2023-10-18T16:07:41.798474+0000","last_undegraded":"2023-10-18T16:07:41.798474+0000","last_fullsized":"2023-10-18T16:07:41.798474+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297320,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074520,"statfs":{"total":96636764160,"available":96332308480,"internally_reserved":0,"allocated":229376,"data_stored":85341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56799999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67100000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297320,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074520,"statfs":{"total":96636764160,"available":96332308480,"internally_reserved":0,"allocated":229376,"data_stored":85341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.318}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48099999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]}]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297320,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074520,"statfs":{"total":96636764160,"available":96332308480,"internally_reserved":0,"allocated":229376,"data_stored":85341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26000000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33100000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.499}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65100000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297320,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074520,"statfs":{"total":96636764160,"available":96332308480,"internally_reserved":0,"allocated":229376,"data_stored":85341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34999999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36899999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60099999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297320,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074520,"statfs":{"total":96636764160,"available":96332308480,"internally_reserved":0,"allocated":229376,"data_stored":85341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42799999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.499}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71699999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68999999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297320,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074520,"statfs":{"total":96636764160,"available":96332308480,"internally_reserved":0,"allocated":229376,"data_stored":85341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34000000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.316}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52500000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59799999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53300000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61299999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297320,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074520,"statfs":{"total":96636764160,"available":96332308480,"internally_reserved":0,"allocated":229376,"data_stored":85341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40600000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.624}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55300000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64100000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297320,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074520,"statfs":{"total":96636764160,"available":96332308480,"internally_reserved":0,"allocated":229376,"data_stored":85341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51200000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67100000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60699999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66100000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.753}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-18T16:07:54.007 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-18T16:07:54.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.145+0000 7f8c98bc1700 1 Processor -- start 2023-10-18T16:07:54.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.145+0000 7f8c98bc1700 1 -- start start 2023-10-18T16:07:54.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.146+0000 7f8c98bc1700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f8c9414d8f0 0x7f8c940c2e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:54.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.146+0000 7f8c98bc1700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8c940a9080 0x7f8c941503b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:54.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.146+0000 7f8c98bc1700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c94150d00 con 0x7f8c941508a0 2023-10-18T16:07:54.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.146+0000 7f8c98bc1700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c940a7190 con 0x7f8c9414d8f0 2023-10-18T16:07:54.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.146+0000 7f8c98bc1700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c940c3360 con 0x7f8c940a9080 2023-10-18T16:07:54.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.146+0000 7f8c9259c700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f8c9414d8f0 0x7f8c940c2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:54.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.146+0000 7f8c89d9b700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8c940a9080 0x7f8c941503b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:54.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.146+0000 7f8c9259c700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f8c9414d8f0 0x7f8c940c2e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.88:60316/0 (socket says 172.21.15.88:60316) 2023-10-18T16:07:54.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.146+0000 7f8c9259c700 1 -- 172.21.15.88:0/3376116909 learned_addr learned my addr 172.21.15.88:0/3376116909 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:54.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.146+0000 7f8c92d9d700 1 --1- 172.21.15.88:0/3376116909 >> v1:172.21.15.151:6789/0 conn(0x7f8c941508a0 0x7f8c940c1f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.88:52664/0 (socket says 172.21.15.88:52664) 2023-10-18T16:07:54.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.147+0000 7f8c9259c700 1 -- 172.21.15.88:0/3376116909 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8c940a9080 msgr2=0x7f8c941503b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:54.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.147+0000 7f8c9259c700 1 --2- 172.21.15.88:0/3376116909 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8c940a9080 0x7f8c941503b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:54.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.147+0000 7f8c9259c700 1 -- 172.21.15.88:0/3376116909 >> v1:172.21.15.151:6789/0 conn(0x7f8c941508a0 legacy=0x7f8c940c1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:54.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.147+0000 7f8c9259c700 1 -- 172.21.15.88:0/3376116909 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c940c3e10 con 0x7f8c9414d8f0 2023-10-18T16:07:54.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.147+0000 7f8c89d9b700 1 --2- 172.21.15.88:0/3376116909 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8c940a9080 0x7f8c941503b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2023-10-18T16:07:54.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.147+0000 7f8c9259c700 1 --2- 172.21.15.88:0/3376116909 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8c9414d8f0 0x7f8c940c2e70 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f8c84006fd0 tx=0x7f8c840090f0).ready entity=mon.1 client_cookie=ce8d6dc51410ee4e server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:54.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.147+0000 7f8c91d9b700 1 -- 172.21.15.88:0/3376116909 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c8400b580 con 0x7f8c9414d8f0 2023-10-18T16:07:54.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.147+0000 7f8c91d9b700 1 -- 172.21.15.88:0/3376116909 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c8400b6e0 con 0x7f8c9414d8f0 2023-10-18T16:07:54.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.147+0000 7f8c91d9b700 1 -- 172.21.15.88:0/3376116909 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c8400ba60 con 0x7f8c9414d8f0 2023-10-18T16:07:54.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.147+0000 7f8c98bc1700 1 -- 172.21.15.88:0/3376116909 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8c9414d8f0 msgr2=0x7f8c940c2e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:54.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.147+0000 7f8c98bc1700 1 --2- 172.21.15.88:0/3376116909 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8c9414d8f0 0x7f8c940c2e70 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f8c84006fd0 tx=0x7f8c840090f0).stop 2023-10-18T16:07:54.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.147+0000 7f8c98bc1700 1 -- 172.21.15.88:0/3376116909 shutdown_connections 2023-10-18T16:07:54.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.147+0000 7f8c98bc1700 1 --2- 172.21.15.88:0/3376116909 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8c9414d8f0 0x7f8c940c2e70 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:54.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.147+0000 7f8c98bc1700 1 --2- 172.21.15.88:0/3376116909 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8c940a9080 0x7f8c941503b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:54.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.147+0000 7f8c98bc1700 1 -- 172.21.15.88:0/3376116909 >> 172.21.15.88:0/3376116909 conn(0x7f8c94069d80 msgr2=0x7f8c9406a190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:54.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.147+0000 7f8c98bc1700 1 -- 172.21.15.88:0/3376116909 shutdown_connections 2023-10-18T16:07:54.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.147+0000 7f8c98bc1700 1 -- 172.21.15.88:0/3376116909 wait complete. 2023-10-18T16:07:54.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.149+0000 7f8c98bc1700 1 Processor -- start 2023-10-18T16:07:54.160 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.149+0000 7f8c98bc1700 1 -- start start 2023-10-18T16:07:54.160 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.149+0000 7f8c98bc1700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c940a9080 0x7f8c9414ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:54.160 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.149+0000 7f8c98bc1700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8c9414d8f0 0x7f8c941610a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:54.160 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.149+0000 7f8c98bc1700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8c941508a0 0x7f8c94147ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:54.160 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.149+0000 7f8c98bc1700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c9414fa50 con 0x7f8c940a9080 2023-10-18T16:07:54.161 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.149+0000 7f8c98bc1700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c941543b0 con 0x7f8c9414d8f0 2023-10-18T16:07:54.161 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.149+0000 7f8c98bc1700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c94161590 con 0x7f8c941508a0 2023-10-18T16:07:54.161 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.149+0000 7f8c9259c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c940a9080 0x7f8c9414ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:54.161 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.149+0000 7f8c9259c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c940a9080 0x7f8c9414ff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59776/0 (socket says 172.21.15.88:59776) 2023-10-18T16:07:54.161 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.149+0000 7f8c9259c700 1 -- 172.21.15.88:0/3133103142 learned_addr learned my addr 172.21.15.88:0/3133103142 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:54.162 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.150+0000 7f8c92d9d700 1 --2- 172.21.15.88:0/3133103142 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8c941508a0 0x7f8c94147ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:54.162 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.150+0000 7f8c89d9b700 1 --2- 172.21.15.88:0/3133103142 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8c9414d8f0 0x7f8c941610a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:54.162 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.150+0000 7f8c9259c700 1 -- 172.21.15.88:0/3133103142 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8c941508a0 msgr2=0x7f8c94147ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:54.162 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.150+0000 7f8c9259c700 1 --2- 172.21.15.88:0/3133103142 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8c941508a0 0x7f8c94147ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:54.162 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.150+0000 7f8c9259c700 1 -- 172.21.15.88:0/3133103142 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8c9414d8f0 msgr2=0x7f8c941610a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:54.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.150+0000 7f8c9259c700 1 --2- 172.21.15.88:0/3133103142 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8c9414d8f0 0x7f8c941610a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:54.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.150+0000 7f8c9259c700 1 -- 172.21.15.88:0/3133103142 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c7c0092a0 con 0x7f8c940a9080 2023-10-18T16:07:54.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.150+0000 7f8c9259c700 1 --2- 172.21.15.88:0/3133103142 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c940a9080 0x7f8c9414ff30 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f8c94150e90 tx=0x7f8c84017000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:54.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.150+0000 7f8c8affd700 1 -- 172.21.15.88:0/3133103142 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c8400b850 con 0x7f8c940a9080 2023-10-18T16:07:54.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.150+0000 7f8c8affd700 1 -- 172.21.15.88:0/3133103142 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c8400b9b0 con 0x7f8c940a9080 2023-10-18T16:07:54.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.150+0000 7f8c98bc1700 1 -- 172.21.15.88:0/3133103142 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c84009370 con 0x7f8c940a9080 2023-10-18T16:07:54.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.150+0000 7f8c8affd700 1 -- 172.21.15.88:0/3133103142 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c8401c670 con 0x7f8c940a9080 2023-10-18T16:07:54.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.150+0000 7f8c98bc1700 1 -- 172.21.15.88:0/3133103142 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c94150570 con 0x7f8c940a9080 2023-10-18T16:07:54.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.151+0000 7f8c98bc1700 1 -- 172.21.15.88:0/3133103142 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c9414fb90 con 0x7f8c940a9080 2023-10-18T16:07:54.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.151+0000 7f8c8affd700 1 -- 172.21.15.88:0/3133103142 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f8c8401ca60 con 0x7f8c940a9080 2023-10-18T16:07:54.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.152+0000 7f8c8affd700 1 --2- 172.21.15.88:0/3133103142 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f8c7405f150 0x7f8c74061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:54.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.152+0000 7f8c8affd700 1 -- 172.21.15.88:0/3133103142 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6121+0+0 (secure 0 0 0) 0x7f8c8407be80 con 0x7f8c940a9080 2023-10-18T16:07:54.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.152+0000 7f8c89d9b700 1 --2- 172.21.15.88:0/3133103142 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f8c7405f150 0x7f8c74061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:54.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.152+0000 7f8c89d9b700 1 --2- 172.21.15.88:0/3133103142 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f8c7405f150 0x7f8c74061610 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f8c7c006d80 tx=0x7f8c7c008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:54.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.156+0000 7f8c8affd700 1 -- 172.21.15.88:0/3133103142 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f8c9414fb90 con 0x7f8c940a9080 2023-10-18T16:07:54.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.315+0000 7f8c98bc1700 1 -- 172.21.15.88:0/3133103142 --> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8c9414fb90 con 0x7f8c7405f150 2023-10-18T16:07:54.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.324+0000 7f8c8affd700 1 -- 172.21.15.88:0/3133103142 <== mgr.4102 v2:172.21.15.151:6832/90916 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+336239 (secure 0 0 0) 0x7f8c9414fb90 con 0x7f8c7405f150 2023-10-18T16:07:54.331 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:07:54.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.327+0000 7f8c98bc1700 1 -- 172.21.15.88:0/3133103142 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f8c7405f150 msgr2=0x7f8c74061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:54.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.327+0000 7f8c98bc1700 1 --2- 172.21.15.88:0/3133103142 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f8c7405f150 0x7f8c74061610 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f8c7c006d80 tx=0x7f8c7c008040).stop 2023-10-18T16:07:54.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.327+0000 7f8c98bc1700 1 -- 172.21.15.88:0/3133103142 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c940a9080 msgr2=0x7f8c9414ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:54.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.327+0000 7f8c98bc1700 1 --2- 172.21.15.88:0/3133103142 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c940a9080 0x7f8c9414ff30 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f8c94150e90 tx=0x7f8c84017000).stop 2023-10-18T16:07:54.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.327+0000 7f8c98bc1700 1 -- 172.21.15.88:0/3133103142 shutdown_connections 2023-10-18T16:07:54.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.327+0000 7f8c98bc1700 1 --2- 172.21.15.88:0/3133103142 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f8c7405f150 0x7f8c74061610 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:54.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.327+0000 7f8c98bc1700 1 --2- 172.21.15.88:0/3133103142 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8c941508a0 0x7f8c94147ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:54.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.327+0000 7f8c98bc1700 1 --2- 172.21.15.88:0/3133103142 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c940a9080 0x7f8c9414ff30 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:54.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.327+0000 7f8c98bc1700 1 --2- 172.21.15.88:0/3133103142 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8c9414d8f0 0x7f8c941610a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:54.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.327+0000 7f8c98bc1700 1 -- 172.21.15.88:0/3133103142 >> 172.21.15.88:0/3133103142 conn(0x7f8c94069d80 msgr2=0x7f8c94149980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:54.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.327+0000 7f8c98bc1700 1 -- 172.21.15.88:0/3133103142 shutdown_connections 2023-10-18T16:07:54.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.327+0000 7f8c98bc1700 1 -- 172.21.15.88:0/3133103142 wait complete. 2023-10-18T16:07:54.337 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2023-10-18T16:07:54.343 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2023-10-18T16:07:52.459463+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":274,"acting":274,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":274,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2378560,"kb_used_data":1792,"kb_used_omap":0,"kb_used_meta":2376704,"kb_avail":752596160,"statfs":{"total":773094113280,"available":770658467840,"internally_reserved":0,"allocated":1835008,"data_stored":682728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2433744896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"4.1d","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.800856+0000","last_change":"2023-10-18T16:07:41.796671+0000","last_active":"2023-10-18T16:07:41.800856+0000","last_peered":"2023-10-18T16:07:41.800856+0000","last_clean":"2023-10-18T16:07:41.800856+0000","last_became_active":"2023-10-18T16:07:41.796451+0000","last_became_peered":"2023-10-18T16:07:41.796451+0000","last_unstale":"2023-10-18T16:07:41.800856+0000","last_undegraded":"2023-10-18T16:07:41.800856+0000","last_fullsized":"2023-10-18T16:07:41.800856+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.794828+0000","last_change":"2023-10-18T16:07:41.442111+0000","last_active":"2023-10-18T16:07:41.794828+0000","last_peered":"2023-10-18T16:07:41.794828+0000","last_clean":"2023-10-18T16:07:41.794828+0000","last_became_active":"2023-10-18T16:07:41.441815+0000","last_became_peered":"2023-10-18T16:07:41.441815+0000","last_unstale":"2023-10-18T16:07:41.794828+0000","last_undegraded":"2023-10-18T16:07:41.794828+0000","last_fullsized":"2023-10-18T16:07:41.794828+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.809204+0000","last_change":"2023-10-18T16:07:41.808517+0000","last_active":"2023-10-18T16:07:41.809204+0000","last_peered":"2023-10-18T16:07:41.809204+0000","last_clean":"2023-10-18T16:07:41.809204+0000","last_became_active":"2023-10-18T16:07:41.808224+0000","last_became_peered":"2023-10-18T16:07:41.808224+0000","last_unstale":"2023-10-18T16:07:41.809204+0000","last_undegraded":"2023-10-18T16:07:41.809204+0000","last_fullsized":"2023-10-18T16:07:41.809204+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.785683+0000","last_change":"2023-10-18T16:07:41.430021+0000","last_active":"2023-10-18T16:07:41.785683+0000","last_peered":"2023-10-18T16:07:41.785683+0000","last_clean":"2023-10-18T16:07:41.785683+0000","last_became_active":"2023-10-18T16:07:41.429781+0000","last_became_peered":"2023-10-18T16:07:41.429781+0000","last_unstale":"2023-10-18T16:07:41.785683+0000","last_undegraded":"2023-10-18T16:07:41.785683+0000","last_fullsized":"2023-10-18T16:07:41.785683+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.808894+0000","last_change":"2023-10-18T16:07:41.807746+0000","last_active":"2023-10-18T16:07:41.808894+0000","last_peered":"2023-10-18T16:07:41.808894+0000","last_clean":"2023-10-18T16:07:41.808894+0000","last_became_active":"2023-10-18T16:07:41.807256+0000","last_became_peered":"2023-10-18T16:07:41.807256+0000","last_unstale":"2023-10-18T16:07:41.808894+0000","last_undegraded":"2023-10-18T16:07:41.808894+0000","last_fullsized":"2023-10-18T16:07:41.808894+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.788944+0000","last_change":"2023-10-18T16:07:41.431122+0000","last_active":"2023-10-18T16:07:41.788944+0000","last_peered":"2023-10-18T16:07:41.788944+0000","last_clean":"2023-10-18T16:07:41.788944+0000","last_became_active":"2023-10-18T16:07:41.430831+0000","last_became_peered":"2023-10-18T16:07:41.430831+0000","last_unstale":"2023-10-18T16:07:41.788944+0000","last_undegraded":"2023-10-18T16:07:41.788944+0000","last_fullsized":"2023-10-18T16:07:41.788944+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.807269+0000","last_change":"2023-10-18T16:07:41.801866+0000","last_active":"2023-10-18T16:07:41.807269+0000","last_peered":"2023-10-18T16:07:41.807269+0000","last_clean":"2023-10-18T16:07:41.807269+0000","last_became_active":"2023-10-18T16:07:41.801602+0000","last_became_peered":"2023-10-18T16:07:41.801602+0000","last_unstale":"2023-10-18T16:07:41.807269+0000","last_undegraded":"2023-10-18T16:07:41.807269+0000","last_fullsized":"2023-10-18T16:07:41.807269+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.790519+0000","last_change":"2023-10-18T16:07:41.442448+0000","last_active":"2023-10-18T16:07:41.790519+0000","last_peered":"2023-10-18T16:07:41.790519+0000","last_clean":"2023-10-18T16:07:41.790519+0000","last_became_active":"2023-10-18T16:07:41.442143+0000","last_became_peered":"2023-10-18T16:07:41.442143+0000","last_unstale":"2023-10-18T16:07:41.790519+0000","last_undegraded":"2023-10-18T16:07:41.790519+0000","last_fullsized":"2023-10-18T16:07:41.790519+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.808841+0000","last_change":"2023-10-18T16:07:41.808302+0000","last_active":"2023-10-18T16:07:41.808841+0000","last_peered":"2023-10-18T16:07:41.808841+0000","last_clean":"2023-10-18T16:07:41.808841+0000","last_became_active":"2023-10-18T16:07:41.808001+0000","last_became_peered":"2023-10-18T16:07:41.808001+0000","last_unstale":"2023-10-18T16:07:41.808841+0000","last_undegraded":"2023-10-18T16:07:41.808841+0000","last_fullsized":"2023-10-18T16:07:41.808841+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.790557+0000","last_change":"2023-10-18T16:07:41.433080+0000","last_active":"2023-10-18T16:07:41.790557+0000","last_peered":"2023-10-18T16:07:41.790557+0000","last_clean":"2023-10-18T16:07:41.790557+0000","last_became_active":"2023-10-18T16:07:41.432714+0000","last_became_peered":"2023-10-18T16:07:41.432714+0000","last_unstale":"2023-10-18T16:07:41.790557+0000","last_undegraded":"2023-10-18T16:07:41.790557+0000","last_fullsized":"2023-10-18T16:07:41.790557+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.803707+0000","last_change":"2023-10-18T16:07:41.797599+0000","last_active":"2023-10-18T16:07:41.803707+0000","last_peered":"2023-10-18T16:07:41.803707+0000","last_clean":"2023-10-18T16:07:41.803707+0000","last_became_active":"2023-10-18T16:07:41.797318+0000","last_became_peered":"2023-10-18T16:07:41.797318+0000","last_unstale":"2023-10-18T16:07:41.803707+0000","last_undegraded":"2023-10-18T16:07:41.803707+0000","last_fullsized":"2023-10-18T16:07:41.803707+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.798325+0000","last_change":"2023-10-18T16:07:41.437425+0000","last_active":"2023-10-18T16:07:41.798325+0000","last_peered":"2023-10-18T16:07:41.798325+0000","last_clean":"2023-10-18T16:07:41.798325+0000","last_became_active":"2023-10-18T16:07:41.437135+0000","last_became_peered":"2023-10-18T16:07:41.437135+0000","last_unstale":"2023-10-18T16:07:41.798325+0000","last_undegraded":"2023-10-18T16:07:41.798325+0000","last_fullsized":"2023-10-18T16:07:41.798325+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.806231+0000","last_change":"2023-10-18T16:07:41.802908+0000","last_active":"2023-10-18T16:07:41.806231+0000","last_peered":"2023-10-18T16:07:41.806231+0000","last_clean":"2023-10-18T16:07:41.806231+0000","last_became_active":"2023-10-18T16:07:41.802644+0000","last_became_peered":"2023-10-18T16:07:41.802644+0000","last_unstale":"2023-10-18T16:07:41.806231+0000","last_undegraded":"2023-10-18T16:07:41.806231+0000","last_fullsized":"2023-10-18T16:07:41.806231+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.796534+0000","last_change":"2023-10-18T16:07:41.446842+0000","last_active":"2023-10-18T16:07:41.796534+0000","last_peered":"2023-10-18T16:07:41.796534+0000","last_clean":"2023-10-18T16:07:41.796534+0000","last_became_active":"2023-10-18T16:07:41.446567+0000","last_became_peered":"2023-10-18T16:07:41.446567+0000","last_unstale":"2023-10-18T16:07:41.796534+0000","last_undegraded":"2023-10-18T16:07:41.796534+0000","last_fullsized":"2023-10-18T16:07:41.796534+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.802559+0000","last_change":"2023-10-18T16:07:41.798736+0000","last_active":"2023-10-18T16:07:41.802559+0000","last_peered":"2023-10-18T16:07:41.802559+0000","last_clean":"2023-10-18T16:07:41.802559+0000","last_became_active":"2023-10-18T16:07:41.798458+0000","last_became_peered":"2023-10-18T16:07:41.798458+0000","last_unstale":"2023-10-18T16:07:41.802559+0000","last_undegraded":"2023-10-18T16:07:41.802559+0000","last_fullsized":"2023-10-18T16:07:41.802559+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.796971+0000","last_change":"2023-10-18T16:07:41.443369+0000","last_active":"2023-10-18T16:07:41.796971+0000","last_peered":"2023-10-18T16:07:41.796971+0000","last_clean":"2023-10-18T16:07:41.796971+0000","last_became_active":"2023-10-18T16:07:41.443064+0000","last_became_peered":"2023-10-18T16:07:41.443064+0000","last_unstale":"2023-10-18T16:07:41.796971+0000","last_undegraded":"2023-10-18T16:07:41.796971+0000","last_fullsized":"2023-10-18T16:07:41.796971+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.809725+0000","last_change":"2023-10-18T16:07:41.808196+0000","last_active":"2023-10-18T16:07:41.809725+0000","last_peered":"2023-10-18T16:07:41.809725+0000","last_clean":"2023-10-18T16:07:41.809725+0000","last_became_active":"2023-10-18T16:07:41.807537+0000","last_became_peered":"2023-10-18T16:07:41.807537+0000","last_unstale":"2023-10-18T16:07:41.809725+0000","last_undegraded":"2023-10-18T16:07:41.809725+0000","last_fullsized":"2023-10-18T16:07:41.809725+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.791192+0000","last_change":"2023-10-18T16:07:41.433498+0000","last_active":"2023-10-18T16:07:41.791192+0000","last_peered":"2023-10-18T16:07:41.791192+0000","last_clean":"2023-10-18T16:07:41.791192+0000","last_became_active":"2023-10-18T16:07:41.433321+0000","last_became_peered":"2023-10-18T16:07:41.433321+0000","last_unstale":"2023-10-18T16:07:41.791192+0000","last_undegraded":"2023-10-18T16:07:41.791192+0000","last_fullsized":"2023-10-18T16:07:41.791192+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.801992+0000","last_change":"2023-10-18T16:07:41.797372+0000","last_active":"2023-10-18T16:07:41.801992+0000","last_peered":"2023-10-18T16:07:41.801992+0000","last_clean":"2023-10-18T16:07:41.801992+0000","last_became_active":"2023-10-18T16:07:41.797108+0000","last_became_peered":"2023-10-18T16:07:41.797108+0000","last_unstale":"2023-10-18T16:07:41.801992+0000","last_undegraded":"2023-10-18T16:07:41.801992+0000","last_fullsized":"2023-10-18T16:07:41.801992+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.799678+0000","last_change":"2023-10-18T16:07:41.445469+0000","last_active":"2023-10-18T16:07:41.799678+0000","last_peered":"2023-10-18T16:07:41.799678+0000","last_clean":"2023-10-18T16:07:41.799678+0000","last_became_active":"2023-10-18T16:07:41.445204+0000","last_became_peered":"2023-10-18T16:07:41.445204+0000","last_unstale":"2023-10-18T16:07:41.799678+0000","last_undegraded":"2023-10-18T16:07:41.799678+0000","last_fullsized":"2023-10-18T16:07:41.799678+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.808614+0000","last_change":"2023-10-18T16:07:41.808046+0000","last_active":"2023-10-18T16:07:41.808614+0000","last_peered":"2023-10-18T16:07:41.808614+0000","last_clean":"2023-10-18T16:07:41.808614+0000","last_became_active":"2023-10-18T16:07:41.807189+0000","last_became_peered":"2023-10-18T16:07:41.807189+0000","last_unstale":"2023-10-18T16:07:41.808614+0000","last_undegraded":"2023-10-18T16:07:41.808614+0000","last_fullsized":"2023-10-18T16:07:41.808614+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.792937+0000","last_change":"2023-10-18T16:07:41.434313+0000","last_active":"2023-10-18T16:07:41.792937+0000","last_peered":"2023-10-18T16:07:41.792937+0000","last_clean":"2023-10-18T16:07:41.792937+0000","last_became_active":"2023-10-18T16:07:41.434022+0000","last_became_peered":"2023-10-18T16:07:41.434022+0000","last_unstale":"2023-10-18T16:07:41.792937+0000","last_undegraded":"2023-10-18T16:07:41.792937+0000","last_fullsized":"2023-10-18T16:07:41.792937+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.806794+0000","last_change":"2023-10-18T16:07:41.800593+0000","last_active":"2023-10-18T16:07:41.806794+0000","last_peered":"2023-10-18T16:07:41.806794+0000","last_clean":"2023-10-18T16:07:41.806794+0000","last_became_active":"2023-10-18T16:07:41.800302+0000","last_became_peered":"2023-10-18T16:07:41.800302+0000","last_unstale":"2023-10-18T16:07:41.806794+0000","last_undegraded":"2023-10-18T16:07:41.806794+0000","last_fullsized":"2023-10-18T16:07:41.806794+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.798209+0000","last_change":"2023-10-18T16:07:41.442977+0000","last_active":"2023-10-18T16:07:41.798209+0000","last_peered":"2023-10-18T16:07:41.798209+0000","last_clean":"2023-10-18T16:07:41.798209+0000","last_became_active":"2023-10-18T16:07:41.442383+0000","last_became_peered":"2023-10-18T16:07:41.442383+0000","last_unstale":"2023-10-18T16:07:41.798209+0000","last_undegraded":"2023-10-18T16:07:41.798209+0000","last_fullsized":"2023-10-18T16:07:41.798209+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.810814+0000","last_change":"2023-10-18T16:07:41.808758+0000","last_active":"2023-10-18T16:07:41.810814+0000","last_peered":"2023-10-18T16:07:41.810814+0000","last_clean":"2023-10-18T16:07:41.810814+0000","last_became_active":"2023-10-18T16:07:41.808475+0000","last_became_peered":"2023-10-18T16:07:41.808475+0000","last_unstale":"2023-10-18T16:07:41.810814+0000","last_undegraded":"2023-10-18T16:07:41.810814+0000","last_fullsized":"2023-10-18T16:07:41.810814+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.798959+0000","last_change":"2023-10-18T16:07:41.442566+0000","last_active":"2023-10-18T16:07:41.798959+0000","last_peered":"2023-10-18T16:07:41.798959+0000","last_clean":"2023-10-18T16:07:41.798959+0000","last_became_active":"2023-10-18T16:07:41.442264+0000","last_became_peered":"2023-10-18T16:07:41.442264+0000","last_unstale":"2023-10-18T16:07:41.798959+0000","last_undegraded":"2023-10-18T16:07:41.798959+0000","last_fullsized":"2023-10-18T16:07:41.798959+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.805558+0000","last_change":"2023-10-18T16:07:41.803408+0000","last_active":"2023-10-18T16:07:41.805558+0000","last_peered":"2023-10-18T16:07:41.805558+0000","last_clean":"2023-10-18T16:07:41.805558+0000","last_became_active":"2023-10-18T16:07:41.803142+0000","last_became_peered":"2023-10-18T16:07:41.803142+0000","last_unstale":"2023-10-18T16:07:41.805558+0000","last_undegraded":"2023-10-18T16:07:41.805558+0000","last_fullsized":"2023-10-18T16:07:41.805558+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.799082+0000","last_change":"2023-10-18T16:07:41.440460+0000","last_active":"2023-10-18T16:07:41.799082+0000","last_peered":"2023-10-18T16:07:41.799082+0000","last_clean":"2023-10-18T16:07:41.799082+0000","last_became_active":"2023-10-18T16:07:41.440162+0000","last_became_peered":"2023-10-18T16:07:41.440162+0000","last_unstale":"2023-10-18T16:07:41.799082+0000","last_undegraded":"2023-10-18T16:07:41.799082+0000","last_fullsized":"2023-10-18T16:07:41.799082+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.807426+0000","last_change":"2023-10-18T16:07:41.804730+0000","last_active":"2023-10-18T16:07:41.807426+0000","last_peered":"2023-10-18T16:07:41.807426+0000","last_clean":"2023-10-18T16:07:41.807426+0000","last_became_active":"2023-10-18T16:07:41.804455+0000","last_became_peered":"2023-10-18T16:07:41.804455+0000","last_unstale":"2023-10-18T16:07:41.807426+0000","last_undegraded":"2023-10-18T16:07:41.807426+0000","last_fullsized":"2023-10-18T16:07:41.807426+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.790384+0000","last_change":"2023-10-18T16:07:41.429156+0000","last_active":"2023-10-18T16:07:41.790384+0000","last_peered":"2023-10-18T16:07:41.790384+0000","last_clean":"2023-10-18T16:07:41.790384+0000","last_became_active":"2023-10-18T16:07:41.428828+0000","last_became_peered":"2023-10-18T16:07:41.428828+0000","last_unstale":"2023-10-18T16:07:41.790384+0000","last_undegraded":"2023-10-18T16:07:41.790384+0000","last_fullsized":"2023-10-18T16:07:41.790384+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.811080+0000","last_change":"2023-10-18T16:07:41.810261+0000","last_active":"2023-10-18T16:07:41.811080+0000","last_peered":"2023-10-18T16:07:41.811080+0000","last_clean":"2023-10-18T16:07:41.811080+0000","last_became_active":"2023-10-18T16:07:41.809938+0000","last_became_peered":"2023-10-18T16:07:41.809938+0000","last_unstale":"2023-10-18T16:07:41.811080+0000","last_undegraded":"2023-10-18T16:07:41.811080+0000","last_fullsized":"2023-10-18T16:07:41.811080+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.788799+0000","last_change":"2023-10-18T16:07:41.436527+0000","last_active":"2023-10-18T16:07:41.788799+0000","last_peered":"2023-10-18T16:07:41.788799+0000","last_clean":"2023-10-18T16:07:41.788799+0000","last_became_active":"2023-10-18T16:07:41.436223+0000","last_became_peered":"2023-10-18T16:07:41.436223+0000","last_unstale":"2023-10-18T16:07:41.788799+0000","last_undegraded":"2023-10-18T16:07:41.788799+0000","last_fullsized":"2023-10-18T16:07:41.788799+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.806186+0000","last_change":"2023-10-18T16:07:41.802220+0000","last_active":"2023-10-18T16:07:41.806186+0000","last_peered":"2023-10-18T16:07:41.806186+0000","last_clean":"2023-10-18T16:07:41.806186+0000","last_became_active":"2023-10-18T16:07:41.801909+0000","last_became_peered":"2023-10-18T16:07:41.801909+0000","last_unstale":"2023-10-18T16:07:41.806186+0000","last_undegraded":"2023-10-18T16:07:41.806186+0000","last_fullsized":"2023-10-18T16:07:41.806186+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.793937+0000","last_change":"2023-10-18T16:07:41.446075+0000","last_active":"2023-10-18T16:07:41.793937+0000","last_peered":"2023-10-18T16:07:41.793937+0000","last_clean":"2023-10-18T16:07:41.793937+0000","last_became_active":"2023-10-18T16:07:41.445817+0000","last_became_peered":"2023-10-18T16:07:41.445817+0000","last_unstale":"2023-10-18T16:07:41.793937+0000","last_undegraded":"2023-10-18T16:07:41.793937+0000","last_fullsized":"2023-10-18T16:07:41.793937+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.806073+0000","last_change":"2023-10-18T16:07:41.802289+0000","last_active":"2023-10-18T16:07:41.806073+0000","last_peered":"2023-10-18T16:07:41.806073+0000","last_clean":"2023-10-18T16:07:41.806073+0000","last_became_active":"2023-10-18T16:07:41.802038+0000","last_became_peered":"2023-10-18T16:07:41.802038+0000","last_unstale":"2023-10-18T16:07:41.806073+0000","last_undegraded":"2023-10-18T16:07:41.806073+0000","last_fullsized":"2023-10-18T16:07:41.806073+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.795284+0000","last_change":"2023-10-18T16:07:41.445650+0000","last_active":"2023-10-18T16:07:41.795284+0000","last_peered":"2023-10-18T16:07:41.795284+0000","last_clean":"2023-10-18T16:07:41.795284+0000","last_became_active":"2023-10-18T16:07:41.445297+0000","last_became_peered":"2023-10-18T16:07:41.445297+0000","last_unstale":"2023-10-18T16:07:41.795284+0000","last_undegraded":"2023-10-18T16:07:41.795284+0000","last_fullsized":"2023-10-18T16:07:41.795284+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.807219+0000","last_change":"2023-10-18T16:07:41.800350+0000","last_active":"2023-10-18T16:07:41.807219+0000","last_peered":"2023-10-18T16:07:41.807219+0000","last_clean":"2023-10-18T16:07:41.807219+0000","last_became_active":"2023-10-18T16:07:41.800095+0000","last_became_peered":"2023-10-18T16:07:41.800095+0000","last_unstale":"2023-10-18T16:07:41.807219+0000","last_undegraded":"2023-10-18T16:07:41.807219+0000","last_fullsized":"2023-10-18T16:07:41.807219+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.789807+0000","last_change":"2023-10-18T16:07:41.429381+0000","last_active":"2023-10-18T16:07:41.789807+0000","last_peered":"2023-10-18T16:07:41.789807+0000","last_clean":"2023-10-18T16:07:41.789807+0000","last_became_active":"2023-10-18T16:07:41.429117+0000","last_became_peered":"2023-10-18T16:07:41.429117+0000","last_unstale":"2023-10-18T16:07:41.789807+0000","last_undegraded":"2023-10-18T16:07:41.789807+0000","last_fullsized":"2023-10-18T16:07:41.789807+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.786761+0000","last_change":"2023-10-18T16:07:37.254194+0000","last_active":"2023-10-18T16:07:41.786761+0000","last_peered":"2023-10-18T16:07:41.786761+0000","last_clean":"2023-10-18T16:07:41.786761+0000","last_became_active":"2023-10-18T16:07:37.253900+0000","last_became_peered":"2023-10-18T16:07:37.253900+0000","last_unstale":"2023-10-18T16:07:41.786761+0000","last_undegraded":"2023-10-18T16:07:41.786761+0000","last_fullsized":"2023-10-18T16:07:41.786761+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:35.522314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:35.522314+0000","last_clean_scrub_stamp":"2023-10-18T16:07:35.522314+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.790385+0000","last_change":"2023-10-18T16:07:41.436498+0000","last_active":"2023-10-18T16:07:41.790385+0000","last_peered":"2023-10-18T16:07:41.790385+0000","last_clean":"2023-10-18T16:07:41.790385+0000","last_became_active":"2023-10-18T16:07:41.436182+0000","last_became_peered":"2023-10-18T16:07:41.436182+0000","last_unstale":"2023-10-18T16:07:41.790385+0000","last_undegraded":"2023-10-18T16:07:41.790385+0000","last_fullsized":"2023-10-18T16:07:41.790385+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.800030+0000","last_change":"2023-10-18T16:07:41.793823+0000","last_active":"2023-10-18T16:07:41.800030+0000","last_peered":"2023-10-18T16:07:41.800030+0000","last_clean":"2023-10-18T16:07:41.800030+0000","last_became_active":"2023-10-18T16:07:41.792600+0000","last_became_peered":"2023-10-18T16:07:41.792600+0000","last_unstale":"2023-10-18T16:07:41.800030+0000","last_undegraded":"2023-10-18T16:07:41.800030+0000","last_fullsized":"2023-10-18T16:07:41.800030+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.790199+0000","last_change":"2023-10-18T16:07:36.544503+0000","last_active":"2023-10-18T16:07:41.790199+0000","last_peered":"2023-10-18T16:07:41.790199+0000","last_clean":"2023-10-18T16:07:41.790199+0000","last_became_active":"2023-10-18T16:07:36.544254+0000","last_became_peered":"2023-10-18T16:07:36.544254+0000","last_unstale":"2023-10-18T16:07:41.790199+0000","last_undegraded":"2023-10-18T16:07:41.790199+0000","last_fullsized":"2023-10-18T16:07:41.790199+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:35.522314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:35.522314+0000","last_clean_scrub_stamp":"2023-10-18T16:07:35.522314+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.795162+0000","last_change":"2023-10-18T16:07:41.437532+0000","last_active":"2023-10-18T16:07:41.795162+0000","last_peered":"2023-10-18T16:07:41.795162+0000","last_clean":"2023-10-18T16:07:41.795162+0000","last_became_active":"2023-10-18T16:07:41.437257+0000","last_became_peered":"2023-10-18T16:07:41.437257+0000","last_unstale":"2023-10-18T16:07:41.795162+0000","last_undegraded":"2023-10-18T16:07:41.795162+0000","last_fullsized":"2023-10-18T16:07:41.795162+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.807397+0000","last_change":"2023-10-18T16:07:41.801719+0000","last_active":"2023-10-18T16:07:41.807397+0000","last_peered":"2023-10-18T16:07:41.807397+0000","last_clean":"2023-10-18T16:07:41.807397+0000","last_became_active":"2023-10-18T16:07:41.801325+0000","last_became_peered":"2023-10-18T16:07:41.801325+0000","last_unstale":"2023-10-18T16:07:41.807397+0000","last_undegraded":"2023-10-18T16:07:41.807397+0000","last_fullsized":"2023-10-18T16:07:41.807397+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.799483+0000","last_change":"2023-10-18T16:07:41.791251+0000","last_active":"2023-10-18T16:07:41.799483+0000","last_peered":"2023-10-18T16:07:41.799483+0000","last_clean":"2023-10-18T16:07:41.799483+0000","last_became_active":"2023-10-18T16:07:41.790992+0000","last_became_peered":"2023-10-18T16:07:41.790992+0000","last_unstale":"2023-10-18T16:07:41.799483+0000","last_undegraded":"2023-10-18T16:07:41.799483+0000","last_fullsized":"2023-10-18T16:07:41.799483+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.795059+0000","last_change":"2023-10-18T16:07:36.542401+0000","last_active":"2023-10-18T16:07:41.795059+0000","last_peered":"2023-10-18T16:07:41.795059+0000","last_clean":"2023-10-18T16:07:41.795059+0000","last_became_active":"2023-10-18T16:07:36.542071+0000","last_became_peered":"2023-10-18T16:07:36.542071+0000","last_unstale":"2023-10-18T16:07:41.795059+0000","last_undegraded":"2023-10-18T16:07:41.795059+0000","last_fullsized":"2023-10-18T16:07:41.795059+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:35.522314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:35.522314+0000","last_clean_scrub_stamp":"2023-10-18T16:07:35.522314+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.785696+0000","last_change":"2023-10-18T16:07:41.428730+0000","last_active":"2023-10-18T16:07:41.785696+0000","last_peered":"2023-10-18T16:07:41.785696+0000","last_clean":"2023-10-18T16:07:41.785696+0000","last_became_active":"2023-10-18T16:07:41.428469+0000","last_became_peered":"2023-10-18T16:07:41.428469+0000","last_unstale":"2023-10-18T16:07:41.785696+0000","last_undegraded":"2023-10-18T16:07:41.785696+0000","last_fullsized":"2023-10-18T16:07:41.785696+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.802624+0000","last_change":"2023-10-18T16:07:41.798241+0000","last_active":"2023-10-18T16:07:41.802624+0000","last_peered":"2023-10-18T16:07:41.802624+0000","last_clean":"2023-10-18T16:07:41.802624+0000","last_became_active":"2023-10-18T16:07:41.797958+0000","last_became_peered":"2023-10-18T16:07:41.797958+0000","last_unstale":"2023-10-18T16:07:41.802624+0000","last_undegraded":"2023-10-18T16:07:41.802624+0000","last_fullsized":"2023-10-18T16:07:41.802624+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.790123+0000","last_change":"2023-10-18T16:07:36.542651+0000","last_active":"2023-10-18T16:07:41.790123+0000","last_peered":"2023-10-18T16:07:41.790123+0000","last_clean":"2023-10-18T16:07:41.790123+0000","last_became_active":"2023-10-18T16:07:36.542306+0000","last_became_peered":"2023-10-18T16:07:36.542306+0000","last_unstale":"2023-10-18T16:07:41.790123+0000","last_undegraded":"2023-10-18T16:07:41.790123+0000","last_fullsized":"2023-10-18T16:07:41.790123+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:35.522314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:35.522314+0000","last_clean_scrub_stamp":"2023-10-18T16:07:35.522314+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.795394+0000","last_change":"2023-10-18T16:07:34.567589+0000","last_active":"2023-10-18T16:07:41.795394+0000","last_peered":"2023-10-18T16:07:41.795394+0000","last_clean":"2023-10-18T16:07:41.795394+0000","last_became_active":"2023-10-18T16:07:34.567301+0000","last_became_peered":"2023-10-18T16:07:34.567301+0000","last_unstale":"2023-10-18T16:07:41.795394+0000","last_undegraded":"2023-10-18T16:07:41.795394+0000","last_fullsized":"2023-10-18T16:07:41.795394+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:33.506699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:33.506699+0000","last_clean_scrub_stamp":"2023-10-18T16:07:33.506699+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.788528+0000","last_change":"2023-10-18T16:07:41.438974+0000","last_active":"2023-10-18T16:07:41.788528+0000","last_peered":"2023-10-18T16:07:41.788528+0000","last_clean":"2023-10-18T16:07:41.788528+0000","last_became_active":"2023-10-18T16:07:41.438710+0000","last_became_peered":"2023-10-18T16:07:41.438710+0000","last_unstale":"2023-10-18T16:07:41.788528+0000","last_undegraded":"2023-10-18T16:07:41.788528+0000","last_fullsized":"2023-10-18T16:07:41.788528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.805329+0000","last_change":"2023-10-18T16:07:41.800880+0000","last_active":"2023-10-18T16:07:41.805329+0000","last_peered":"2023-10-18T16:07:41.805329+0000","last_clean":"2023-10-18T16:07:41.805329+0000","last_became_active":"2023-10-18T16:07:41.800478+0000","last_became_peered":"2023-10-18T16:07:41.800478+0000","last_unstale":"2023-10-18T16:07:41.805329+0000","last_undegraded":"2023-10-18T16:07:41.805329+0000","last_fullsized":"2023-10-18T16:07:41.805329+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.795831+0000","last_change":"2023-10-18T16:07:36.542454+0000","last_active":"2023-10-18T16:07:41.795831+0000","last_peered":"2023-10-18T16:07:41.795831+0000","last_clean":"2023-10-18T16:07:41.795831+0000","last_became_active":"2023-10-18T16:07:36.542134+0000","last_became_peered":"2023-10-18T16:07:36.542134+0000","last_unstale":"2023-10-18T16:07:41.795831+0000","last_undegraded":"2023-10-18T16:07:41.795831+0000","last_fullsized":"2023-10-18T16:07:41.795831+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:35.522314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:35.522314+0000","last_clean_scrub_stamp":"2023-10-18T16:07:35.522314+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.790424+0000","last_change":"2023-10-18T16:07:41.445653+0000","last_active":"2023-10-18T16:07:41.790424+0000","last_peered":"2023-10-18T16:07:41.790424+0000","last_clean":"2023-10-18T16:07:41.790424+0000","last_became_active":"2023-10-18T16:07:41.445399+0000","last_became_peered":"2023-10-18T16:07:41.445399+0000","last_unstale":"2023-10-18T16:07:41.790424+0000","last_undegraded":"2023-10-18T16:07:41.790424+0000","last_fullsized":"2023-10-18T16:07:41.790424+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.798965+0000","last_change":"2023-10-18T16:07:37.254490+0000","last_active":"2023-10-18T16:07:41.798965+0000","last_peered":"2023-10-18T16:07:41.798965+0000","last_clean":"2023-10-18T16:07:41.798965+0000","last_became_active":"2023-10-18T16:07:37.254182+0000","last_became_peered":"2023-10-18T16:07:37.254182+0000","last_unstale":"2023-10-18T16:07:41.798965+0000","last_undegraded":"2023-10-18T16:07:41.798965+0000","last_fullsized":"2023-10-18T16:07:41.798965+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:35.522314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:35.522314+0000","last_clean_scrub_stamp":"2023-10-18T16:07:35.522314+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.787199+0000","last_change":"2023-10-18T16:07:41.428721+0000","last_active":"2023-10-18T16:07:41.787199+0000","last_peered":"2023-10-18T16:07:41.787199+0000","last_clean":"2023-10-18T16:07:41.787199+0000","last_became_active":"2023-10-18T16:07:41.428460+0000","last_became_peered":"2023-10-18T16:07:41.428460+0000","last_unstale":"2023-10-18T16:07:41.787199+0000","last_undegraded":"2023-10-18T16:07:41.787199+0000","last_fullsized":"2023-10-18T16:07:41.787199+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.805475+0000","last_change":"2023-10-18T16:07:41.801499+0000","last_active":"2023-10-18T16:07:41.805475+0000","last_peered":"2023-10-18T16:07:41.805475+0000","last_clean":"2023-10-18T16:07:41.805475+0000","last_became_active":"2023-10-18T16:07:41.800625+0000","last_became_peered":"2023-10-18T16:07:41.800625+0000","last_unstale":"2023-10-18T16:07:41.805475+0000","last_undegraded":"2023-10-18T16:07:41.805475+0000","last_fullsized":"2023-10-18T16:07:41.805475+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.789569+0000","last_change":"2023-10-18T16:07:41.435428+0000","last_active":"2023-10-18T16:07:41.789569+0000","last_peered":"2023-10-18T16:07:41.789569+0000","last_clean":"2023-10-18T16:07:41.789569+0000","last_became_active":"2023-10-18T16:07:41.435124+0000","last_became_peered":"2023-10-18T16:07:41.435124+0000","last_unstale":"2023-10-18T16:07:41.789569+0000","last_undegraded":"2023-10-18T16:07:41.789569+0000","last_fullsized":"2023-10-18T16:07:41.789569+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.785797+0000","last_change":"2023-10-18T16:07:36.542463+0000","last_active":"2023-10-18T16:07:41.785797+0000","last_peered":"2023-10-18T16:07:41.785797+0000","last_clean":"2023-10-18T16:07:41.785797+0000","last_became_active":"2023-10-18T16:07:36.542090+0000","last_became_peered":"2023-10-18T16:07:36.542090+0000","last_unstale":"2023-10-18T16:07:41.785797+0000","last_undegraded":"2023-10-18T16:07:41.785797+0000","last_fullsized":"2023-10-18T16:07:41.785797+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:35.522314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:35.522314+0000","last_clean_scrub_stamp":"2023-10-18T16:07:35.522314+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.807288+0000","last_change":"2023-10-18T16:07:41.802294+0000","last_active":"2023-10-18T16:07:41.807288+0000","last_peered":"2023-10-18T16:07:41.807288+0000","last_clean":"2023-10-18T16:07:41.807288+0000","last_became_active":"2023-10-18T16:07:41.801983+0000","last_became_peered":"2023-10-18T16:07:41.801983+0000","last_unstale":"2023-10-18T16:07:41.807288+0000","last_undegraded":"2023-10-18T16:07:41.807288+0000","last_fullsized":"2023-10-18T16:07:41.807288+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.793283+0000","last_change":"2023-10-18T16:07:36.540468+0000","last_active":"2023-10-18T16:07:41.793283+0000","last_peered":"2023-10-18T16:07:41.793283+0000","last_clean":"2023-10-18T16:07:41.793283+0000","last_became_active":"2023-10-18T16:07:36.540145+0000","last_became_peered":"2023-10-18T16:07:36.540145+0000","last_unstale":"2023-10-18T16:07:41.793283+0000","last_undegraded":"2023-10-18T16:07:41.793283+0000","last_fullsized":"2023-10-18T16:07:41.793283+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:35.522314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:35.522314+0000","last_clean_scrub_stamp":"2023-10-18T16:07:35.522314+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.791238+0000","last_change":"2023-10-18T16:07:41.431117+0000","last_active":"2023-10-18T16:07:41.791238+0000","last_peered":"2023-10-18T16:07:41.791238+0000","last_clean":"2023-10-18T16:07:41.791238+0000","last_became_active":"2023-10-18T16:07:41.430838+0000","last_became_peered":"2023-10-18T16:07:41.430838+0000","last_unstale":"2023-10-18T16:07:41.791238+0000","last_undegraded":"2023-10-18T16:07:41.791238+0000","last_fullsized":"2023-10-18T16:07:41.791238+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.806469+0000","last_change":"2023-10-18T16:07:41.803250+0000","last_active":"2023-10-18T16:07:41.806469+0000","last_peered":"2023-10-18T16:07:41.806469+0000","last_clean":"2023-10-18T16:07:41.806469+0000","last_became_active":"2023-10-18T16:07:41.802930+0000","last_became_peered":"2023-10-18T16:07:41.802930+0000","last_unstale":"2023-10-18T16:07:41.806469+0000","last_undegraded":"2023-10-18T16:07:41.806469+0000","last_fullsized":"2023-10-18T16:07:41.806469+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.807328+0000","last_change":"2023-10-18T16:07:41.803179+0000","last_active":"2023-10-18T16:07:41.807328+0000","last_peered":"2023-10-18T16:07:41.807328+0000","last_clean":"2023-10-18T16:07:41.807328+0000","last_became_active":"2023-10-18T16:07:41.802934+0000","last_became_peered":"2023-10-18T16:07:41.802934+0000","last_unstale":"2023-10-18T16:07:41.807328+0000","last_undegraded":"2023-10-18T16:07:41.807328+0000","last_fullsized":"2023-10-18T16:07:41.807328+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.792656+0000","last_change":"2023-10-18T16:07:41.443157+0000","last_active":"2023-10-18T16:07:41.792656+0000","last_peered":"2023-10-18T16:07:41.792656+0000","last_clean":"2023-10-18T16:07:41.792656+0000","last_became_active":"2023-10-18T16:07:41.442898+0000","last_became_peered":"2023-10-18T16:07:41.442898+0000","last_unstale":"2023-10-18T16:07:41.792656+0000","last_undegraded":"2023-10-18T16:07:41.792656+0000","last_fullsized":"2023-10-18T16:07:41.792656+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.807624+0000","last_change":"2023-10-18T16:07:41.804233+0000","last_active":"2023-10-18T16:07:41.807624+0000","last_peered":"2023-10-18T16:07:41.807624+0000","last_clean":"2023-10-18T16:07:41.807624+0000","last_became_active":"2023-10-18T16:07:41.803966+0000","last_became_peered":"2023-10-18T16:07:41.803966+0000","last_unstale":"2023-10-18T16:07:41.807624+0000","last_undegraded":"2023-10-18T16:07:41.807624+0000","last_fullsized":"2023-10-18T16:07:41.807624+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.790546+0000","last_change":"2023-10-18T16:07:41.443533+0000","last_active":"2023-10-18T16:07:41.790546+0000","last_peered":"2023-10-18T16:07:41.790546+0000","last_clean":"2023-10-18T16:07:41.790546+0000","last_became_active":"2023-10-18T16:07:41.443276+0000","last_became_peered":"2023-10-18T16:07:41.443276+0000","last_unstale":"2023-10-18T16:07:41.790546+0000","last_undegraded":"2023-10-18T16:07:41.790546+0000","last_fullsized":"2023-10-18T16:07:41.790546+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.805621+0000","last_change":"2023-10-18T16:07:41.799449+0000","last_active":"2023-10-18T16:07:41.805621+0000","last_peered":"2023-10-18T16:07:41.805621+0000","last_clean":"2023-10-18T16:07:41.805621+0000","last_became_active":"2023-10-18T16:07:41.799177+0000","last_became_peered":"2023-10-18T16:07:41.799177+0000","last_unstale":"2023-10-18T16:07:41.805621+0000","last_undegraded":"2023-10-18T16:07:41.805621+0000","last_fullsized":"2023-10-18T16:07:41.805621+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.790477+0000","last_change":"2023-10-18T16:07:41.443654+0000","last_active":"2023-10-18T16:07:41.790477+0000","last_peered":"2023-10-18T16:07:41.790477+0000","last_clean":"2023-10-18T16:07:41.790477+0000","last_became_active":"2023-10-18T16:07:41.443453+0000","last_became_peered":"2023-10-18T16:07:41.443453+0000","last_unstale":"2023-10-18T16:07:41.790477+0000","last_undegraded":"2023-10-18T16:07:41.790477+0000","last_fullsized":"2023-10-18T16:07:41.790477+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.802526+0000","last_change":"2023-10-18T16:07:41.798748+0000","last_active":"2023-10-18T16:07:41.802526+0000","last_peered":"2023-10-18T16:07:41.802526+0000","last_clean":"2023-10-18T16:07:41.802526+0000","last_became_active":"2023-10-18T16:07:41.798479+0000","last_became_peered":"2023-10-18T16:07:41.798479+0000","last_unstale":"2023-10-18T16:07:41.802526+0000","last_undegraded":"2023-10-18T16:07:41.802526+0000","last_fullsized":"2023-10-18T16:07:41.802526+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.788543+0000","last_change":"2023-10-18T16:07:41.438072+0000","last_active":"2023-10-18T16:07:41.788543+0000","last_peered":"2023-10-18T16:07:41.788543+0000","last_clean":"2023-10-18T16:07:41.788543+0000","last_became_active":"2023-10-18T16:07:41.437665+0000","last_became_peered":"2023-10-18T16:07:41.437665+0000","last_unstale":"2023-10-18T16:07:41.788543+0000","last_undegraded":"2023-10-18T16:07:41.788543+0000","last_fullsized":"2023-10-18T16:07:41.788543+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.806103+0000","last_change":"2023-10-18T16:07:41.800789+0000","last_active":"2023-10-18T16:07:41.806103+0000","last_peered":"2023-10-18T16:07:41.806103+0000","last_clean":"2023-10-18T16:07:41.806103+0000","last_became_active":"2023-10-18T16:07:41.800336+0000","last_became_peered":"2023-10-18T16:07:41.800336+0000","last_unstale":"2023-10-18T16:07:41.806103+0000","last_undegraded":"2023-10-18T16:07:41.806103+0000","last_fullsized":"2023-10-18T16:07:41.806103+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.790623+0000","last_change":"2023-10-18T16:07:41.439886+0000","last_active":"2023-10-18T16:07:41.790623+0000","last_peered":"2023-10-18T16:07:41.790623+0000","last_clean":"2023-10-18T16:07:41.790623+0000","last_became_active":"2023-10-18T16:07:41.439664+0000","last_became_peered":"2023-10-18T16:07:41.439664+0000","last_unstale":"2023-10-18T16:07:41.790623+0000","last_undegraded":"2023-10-18T16:07:41.790623+0000","last_fullsized":"2023-10-18T16:07:41.790623+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.803656+0000","last_change":"2023-10-18T16:07:41.797792+0000","last_active":"2023-10-18T16:07:41.803656+0000","last_peered":"2023-10-18T16:07:41.803656+0000","last_clean":"2023-10-18T16:07:41.803656+0000","last_became_active":"2023-10-18T16:07:41.797498+0000","last_became_peered":"2023-10-18T16:07:41.797498+0000","last_unstale":"2023-10-18T16:07:41.803656+0000","last_undegraded":"2023-10-18T16:07:41.803656+0000","last_fullsized":"2023-10-18T16:07:41.803656+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.793059+0000","last_change":"2023-10-18T16:07:41.434136+0000","last_active":"2023-10-18T16:07:41.793059+0000","last_peered":"2023-10-18T16:07:41.793059+0000","last_clean":"2023-10-18T16:07:41.793059+0000","last_became_active":"2023-10-18T16:07:41.433862+0000","last_became_peered":"2023-10-18T16:07:41.433862+0000","last_unstale":"2023-10-18T16:07:41.793059+0000","last_undegraded":"2023-10-18T16:07:41.793059+0000","last_fullsized":"2023-10-18T16:07:41.793059+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.801349+0000","last_change":"2023-10-18T16:07:41.797301+0000","last_active":"2023-10-18T16:07:41.801349+0000","last_peered":"2023-10-18T16:07:41.801349+0000","last_clean":"2023-10-18T16:07:41.801349+0000","last_became_active":"2023-10-18T16:07:41.797029+0000","last_became_peered":"2023-10-18T16:07:41.797029+0000","last_unstale":"2023-10-18T16:07:41.801349+0000","last_undegraded":"2023-10-18T16:07:41.801349+0000","last_fullsized":"2023-10-18T16:07:41.801349+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.795552+0000","last_change":"2023-10-18T16:07:41.426435+0000","last_active":"2023-10-18T16:07:41.795552+0000","last_peered":"2023-10-18T16:07:41.795552+0000","last_clean":"2023-10-18T16:07:41.795552+0000","last_became_active":"2023-10-18T16:07:41.426169+0000","last_became_peered":"2023-10-18T16:07:41.426169+0000","last_unstale":"2023-10-18T16:07:41.795552+0000","last_undegraded":"2023-10-18T16:07:41.795552+0000","last_fullsized":"2023-10-18T16:07:41.795552+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.809706+0000","last_change":"2023-10-18T16:07:41.808005+0000","last_active":"2023-10-18T16:07:41.809706+0000","last_peered":"2023-10-18T16:07:41.809706+0000","last_clean":"2023-10-18T16:07:41.809706+0000","last_became_active":"2023-10-18T16:07:41.807155+0000","last_became_peered":"2023-10-18T16:07:41.807155+0000","last_unstale":"2023-10-18T16:07:41.809706+0000","last_undegraded":"2023-10-18T16:07:41.809706+0000","last_fullsized":"2023-10-18T16:07:41.809706+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.790085+0000","last_change":"2023-10-18T16:07:41.437945+0000","last_active":"2023-10-18T16:07:41.790085+0000","last_peered":"2023-10-18T16:07:41.790085+0000","last_clean":"2023-10-18T16:07:41.790085+0000","last_became_active":"2023-10-18T16:07:41.437118+0000","last_became_peered":"2023-10-18T16:07:41.437118+0000","last_unstale":"2023-10-18T16:07:41.790085+0000","last_undegraded":"2023-10-18T16:07:41.790085+0000","last_fullsized":"2023-10-18T16:07:41.790085+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.807377+0000","last_change":"2023-10-18T16:07:41.801600+0000","last_active":"2023-10-18T16:07:41.807377+0000","last_peered":"2023-10-18T16:07:41.807377+0000","last_clean":"2023-10-18T16:07:41.807377+0000","last_became_active":"2023-10-18T16:07:41.801002+0000","last_became_peered":"2023-10-18T16:07:41.801002+0000","last_unstale":"2023-10-18T16:07:41.807377+0000","last_undegraded":"2023-10-18T16:07:41.807377+0000","last_fullsized":"2023-10-18T16:07:41.807377+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.792471+0000","last_change":"2023-10-18T16:07:41.432368+0000","last_active":"2023-10-18T16:07:41.792471+0000","last_peered":"2023-10-18T16:07:41.792471+0000","last_clean":"2023-10-18T16:07:41.792471+0000","last_became_active":"2023-10-18T16:07:41.431923+0000","last_became_peered":"2023-10-18T16:07:41.431923+0000","last_unstale":"2023-10-18T16:07:41.792471+0000","last_undegraded":"2023-10-18T16:07:41.792471+0000","last_fullsized":"2023-10-18T16:07:41.792471+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.810702+0000","last_change":"2023-10-18T16:07:41.809593+0000","last_active":"2023-10-18T16:07:41.810702+0000","last_peered":"2023-10-18T16:07:41.810702+0000","last_clean":"2023-10-18T16:07:41.810702+0000","last_became_active":"2023-10-18T16:07:41.809309+0000","last_became_peered":"2023-10-18T16:07:41.809309+0000","last_unstale":"2023-10-18T16:07:41.810702+0000","last_undegraded":"2023-10-18T16:07:41.810702+0000","last_fullsized":"2023-10-18T16:07:41.810702+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.789348+0000","last_change":"2023-10-18T16:07:41.443916+0000","last_active":"2023-10-18T16:07:41.789348+0000","last_peered":"2023-10-18T16:07:41.789348+0000","last_clean":"2023-10-18T16:07:41.789348+0000","last_became_active":"2023-10-18T16:07:41.443695+0000","last_became_peered":"2023-10-18T16:07:41.443695+0000","last_unstale":"2023-10-18T16:07:41.789348+0000","last_undegraded":"2023-10-18T16:07:41.789348+0000","last_fullsized":"2023-10-18T16:07:41.789348+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.806255+0000","last_change":"2023-10-18T16:07:41.803821+0000","last_active":"2023-10-18T16:07:41.806255+0000","last_peered":"2023-10-18T16:07:41.806255+0000","last_clean":"2023-10-18T16:07:41.806255+0000","last_became_active":"2023-10-18T16:07:41.803540+0000","last_became_peered":"2023-10-18T16:07:41.803540+0000","last_unstale":"2023-10-18T16:07:41.806255+0000","last_undegraded":"2023-10-18T16:07:41.806255+0000","last_fullsized":"2023-10-18T16:07:41.806255+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.786238+0000","last_change":"2023-10-18T16:07:41.429591+0000","last_active":"2023-10-18T16:07:41.786238+0000","last_peered":"2023-10-18T16:07:41.786238+0000","last_clean":"2023-10-18T16:07:41.786238+0000","last_became_active":"2023-10-18T16:07:41.429290+0000","last_became_peered":"2023-10-18T16:07:41.429290+0000","last_unstale":"2023-10-18T16:07:41.786238+0000","last_undegraded":"2023-10-18T16:07:41.786238+0000","last_fullsized":"2023-10-18T16:07:41.786238+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.799825+0000","last_change":"2023-10-18T16:07:41.795679+0000","last_active":"2023-10-18T16:07:41.799825+0000","last_peered":"2023-10-18T16:07:41.799825+0000","last_clean":"2023-10-18T16:07:41.799825+0000","last_became_active":"2023-10-18T16:07:41.795413+0000","last_became_peered":"2023-10-18T16:07:41.795413+0000","last_unstale":"2023-10-18T16:07:41.799825+0000","last_undegraded":"2023-10-18T16:07:41.799825+0000","last_fullsized":"2023-10-18T16:07:41.799825+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.793196+0000","last_change":"2023-10-18T16:07:41.436812+0000","last_active":"2023-10-18T16:07:41.793196+0000","last_peered":"2023-10-18T16:07:41.793196+0000","last_clean":"2023-10-18T16:07:41.793196+0000","last_became_active":"2023-10-18T16:07:41.436541+0000","last_became_peered":"2023-10-18T16:07:41.436541+0000","last_unstale":"2023-10-18T16:07:41.793196+0000","last_undegraded":"2023-10-18T16:07:41.793196+0000","last_fullsized":"2023-10-18T16:07:41.793196+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.806410+0000","last_change":"2023-10-18T16:07:41.803115+0000","last_active":"2023-10-18T16:07:41.806410+0000","last_peered":"2023-10-18T16:07:41.806410+0000","last_clean":"2023-10-18T16:07:41.806410+0000","last_became_active":"2023-10-18T16:07:41.802828+0000","last_became_peered":"2023-10-18T16:07:41.802828+0000","last_unstale":"2023-10-18T16:07:41.806410+0000","last_undegraded":"2023-10-18T16:07:41.806410+0000","last_fullsized":"2023-10-18T16:07:41.806410+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.789662+0000","last_change":"2023-10-18T16:07:41.443476+0000","last_active":"2023-10-18T16:07:41.789662+0000","last_peered":"2023-10-18T16:07:41.789662+0000","last_clean":"2023-10-18T16:07:41.789662+0000","last_became_active":"2023-10-18T16:07:41.443295+0000","last_became_peered":"2023-10-18T16:07:41.443295+0000","last_unstale":"2023-10-18T16:07:41.789662+0000","last_undegraded":"2023-10-18T16:07:41.789662+0000","last_fullsized":"2023-10-18T16:07:41.789662+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.805189+0000","last_change":"2023-10-18T16:07:41.800058+0000","last_active":"2023-10-18T16:07:41.805189+0000","last_peered":"2023-10-18T16:07:41.805189+0000","last_clean":"2023-10-18T16:07:41.805189+0000","last_became_active":"2023-10-18T16:07:41.799452+0000","last_became_peered":"2023-10-18T16:07:41.799452+0000","last_unstale":"2023-10-18T16:07:41.805189+0000","last_undegraded":"2023-10-18T16:07:41.805189+0000","last_fullsized":"2023-10-18T16:07:41.805189+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.789303+0000","last_change":"2023-10-18T16:07:41.440724+0000","last_active":"2023-10-18T16:07:41.789303+0000","last_peered":"2023-10-18T16:07:41.789303+0000","last_clean":"2023-10-18T16:07:41.789303+0000","last_became_active":"2023-10-18T16:07:41.440435+0000","last_became_peered":"2023-10-18T16:07:41.440435+0000","last_unstale":"2023-10-18T16:07:41.789303+0000","last_undegraded":"2023-10-18T16:07:41.789303+0000","last_fullsized":"2023-10-18T16:07:41.789303+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.806559+0000","last_change":"2023-10-18T16:07:41.804438+0000","last_active":"2023-10-18T16:07:41.806559+0000","last_peered":"2023-10-18T16:07:41.806559+0000","last_clean":"2023-10-18T16:07:41.806559+0000","last_became_active":"2023-10-18T16:07:41.804126+0000","last_became_peered":"2023-10-18T16:07:41.804126+0000","last_unstale":"2023-10-18T16:07:41.806559+0000","last_undegraded":"2023-10-18T16:07:41.806559+0000","last_fullsized":"2023-10-18T16:07:41.806559+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.788864+0000","last_change":"2023-10-18T16:07:41.447812+0000","last_active":"2023-10-18T16:07:41.788864+0000","last_peered":"2023-10-18T16:07:41.788864+0000","last_clean":"2023-10-18T16:07:41.788864+0000","last_became_active":"2023-10-18T16:07:41.447479+0000","last_became_peered":"2023-10-18T16:07:41.447479+0000","last_unstale":"2023-10-18T16:07:41.788864+0000","last_undegraded":"2023-10-18T16:07:41.788864+0000","last_fullsized":"2023-10-18T16:07:41.788864+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.804861+0000","last_change":"2023-10-18T16:07:41.801084+0000","last_active":"2023-10-18T16:07:41.804861+0000","last_peered":"2023-10-18T16:07:41.804861+0000","last_clean":"2023-10-18T16:07:41.804861+0000","last_became_active":"2023-10-18T16:07:41.800783+0000","last_became_peered":"2023-10-18T16:07:41.800783+0000","last_unstale":"2023-10-18T16:07:41.804861+0000","last_undegraded":"2023-10-18T16:07:41.804861+0000","last_fullsized":"2023-10-18T16:07:41.804861+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.790315+0000","last_change":"2023-10-18T16:07:41.437755+0000","last_active":"2023-10-18T16:07:41.790315+0000","last_peered":"2023-10-18T16:07:41.790315+0000","last_clean":"2023-10-18T16:07:41.790315+0000","last_became_active":"2023-10-18T16:07:41.436781+0000","last_became_peered":"2023-10-18T16:07:41.436781+0000","last_unstale":"2023-10-18T16:07:41.790315+0000","last_undegraded":"2023-10-18T16:07:41.790315+0000","last_fullsized":"2023-10-18T16:07:41.790315+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.810938+0000","last_change":"2023-10-18T16:07:41.808318+0000","last_active":"2023-10-18T16:07:41.810938+0000","last_peered":"2023-10-18T16:07:41.810938+0000","last_clean":"2023-10-18T16:07:41.810938+0000","last_became_active":"2023-10-18T16:07:41.807983+0000","last_became_peered":"2023-10-18T16:07:41.807983+0000","last_unstale":"2023-10-18T16:07:41.810938+0000","last_undegraded":"2023-10-18T16:07:41.810938+0000","last_fullsized":"2023-10-18T16:07:41.810938+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.792563+0000","last_change":"2023-10-18T16:07:41.442119+0000","last_active":"2023-10-18T16:07:41.792563+0000","last_peered":"2023-10-18T16:07:41.792563+0000","last_clean":"2023-10-18T16:07:41.792563+0000","last_became_active":"2023-10-18T16:07:41.441816+0000","last_became_peered":"2023-10-18T16:07:41.441816+0000","last_unstale":"2023-10-18T16:07:41.792563+0000","last_undegraded":"2023-10-18T16:07:41.792563+0000","last_fullsized":"2023-10-18T16:07:41.792563+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.810523+0000","last_change":"2023-10-18T16:07:41.809378+0000","last_active":"2023-10-18T16:07:41.810523+0000","last_peered":"2023-10-18T16:07:41.810523+0000","last_clean":"2023-10-18T16:07:41.810523+0000","last_became_active":"2023-10-18T16:07:41.809102+0000","last_became_peered":"2023-10-18T16:07:41.809102+0000","last_unstale":"2023-10-18T16:07:41.810523+0000","last_undegraded":"2023-10-18T16:07:41.810523+0000","last_fullsized":"2023-10-18T16:07:41.810523+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.793639+0000","last_change":"2023-10-18T16:07:41.442792+0000","last_active":"2023-10-18T16:07:41.793639+0000","last_peered":"2023-10-18T16:07:41.793639+0000","last_clean":"2023-10-18T16:07:41.793639+0000","last_became_active":"2023-10-18T16:07:41.442506+0000","last_became_peered":"2023-10-18T16:07:41.442506+0000","last_unstale":"2023-10-18T16:07:41.793639+0000","last_undegraded":"2023-10-18T16:07:41.793639+0000","last_fullsized":"2023-10-18T16:07:41.793639+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.807999+0000","last_change":"2023-10-18T16:07:41.804704+0000","last_active":"2023-10-18T16:07:41.807999+0000","last_peered":"2023-10-18T16:07:41.807999+0000","last_clean":"2023-10-18T16:07:41.807999+0000","last_became_active":"2023-10-18T16:07:41.804437+0000","last_became_peered":"2023-10-18T16:07:41.804437+0000","last_unstale":"2023-10-18T16:07:41.807999+0000","last_undegraded":"2023-10-18T16:07:41.807999+0000","last_fullsized":"2023-10-18T16:07:41.807999+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.798211+0000","last_change":"2023-10-18T16:07:41.436171+0000","last_active":"2023-10-18T16:07:41.798211+0000","last_peered":"2023-10-18T16:07:41.798211+0000","last_clean":"2023-10-18T16:07:41.798211+0000","last_became_active":"2023-10-18T16:07:41.435844+0000","last_became_peered":"2023-10-18T16:07:41.435844+0000","last_unstale":"2023-10-18T16:07:41.798211+0000","last_undegraded":"2023-10-18T16:07:41.798211+0000","last_fullsized":"2023-10-18T16:07:41.798211+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.802055+0000","last_change":"2023-10-18T16:07:41.796556+0000","last_active":"2023-10-18T16:07:41.802055+0000","last_peered":"2023-10-18T16:07:41.802055+0000","last_clean":"2023-10-18T16:07:41.802055+0000","last_became_active":"2023-10-18T16:07:41.796291+0000","last_became_peered":"2023-10-18T16:07:41.796291+0000","last_unstale":"2023-10-18T16:07:41.802055+0000","last_undegraded":"2023-10-18T16:07:41.802055+0000","last_fullsized":"2023-10-18T16:07:41.802055+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.795432+0000","last_change":"2023-10-18T16:07:41.434332+0000","last_active":"2023-10-18T16:07:41.795432+0000","last_peered":"2023-10-18T16:07:41.795432+0000","last_clean":"2023-10-18T16:07:41.795432+0000","last_became_active":"2023-10-18T16:07:41.434076+0000","last_became_peered":"2023-10-18T16:07:41.434076+0000","last_unstale":"2023-10-18T16:07:41.795432+0000","last_undegraded":"2023-10-18T16:07:41.795432+0000","last_fullsized":"2023-10-18T16:07:41.795432+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.808487+0000","last_change":"2023-10-18T16:07:41.806379+0000","last_active":"2023-10-18T16:07:41.808487+0000","last_peered":"2023-10-18T16:07:41.808487+0000","last_clean":"2023-10-18T16:07:41.808487+0000","last_became_active":"2023-10-18T16:07:41.806018+0000","last_became_peered":"2023-10-18T16:07:41.806018+0000","last_unstale":"2023-10-18T16:07:41.808487+0000","last_undegraded":"2023-10-18T16:07:41.808487+0000","last_fullsized":"2023-10-18T16:07:41.808487+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.790761+0000","last_change":"2023-10-18T16:07:41.438688+0000","last_active":"2023-10-18T16:07:41.790761+0000","last_peered":"2023-10-18T16:07:41.790761+0000","last_clean":"2023-10-18T16:07:41.790761+0000","last_became_active":"2023-10-18T16:07:41.438405+0000","last_became_peered":"2023-10-18T16:07:41.438405+0000","last_unstale":"2023-10-18T16:07:41.790761+0000","last_undegraded":"2023-10-18T16:07:41.790761+0000","last_fullsized":"2023-10-18T16:07:41.790761+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.805209+0000","last_change":"2023-10-18T16:07:41.800230+0000","last_active":"2023-10-18T16:07:41.805209+0000","last_peered":"2023-10-18T16:07:41.805209+0000","last_clean":"2023-10-18T16:07:41.805209+0000","last_became_active":"2023-10-18T16:07:41.799787+0000","last_became_peered":"2023-10-18T16:07:41.799787+0000","last_unstale":"2023-10-18T16:07:41.805209+0000","last_undegraded":"2023-10-18T16:07:41.805209+0000","last_fullsized":"2023-10-18T16:07:41.805209+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.789370+0000","last_change":"2023-10-18T16:07:41.440794+0000","last_active":"2023-10-18T16:07:41.789370+0000","last_peered":"2023-10-18T16:07:41.789370+0000","last_clean":"2023-10-18T16:07:41.789370+0000","last_became_active":"2023-10-18T16:07:41.440412+0000","last_became_peered":"2023-10-18T16:07:41.440412+0000","last_unstale":"2023-10-18T16:07:41.789370+0000","last_undegraded":"2023-10-18T16:07:41.789370+0000","last_fullsized":"2023-10-18T16:07:41.789370+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.806876+0000","last_change":"2023-10-18T16:07:41.801402+0000","last_active":"2023-10-18T16:07:41.806876+0000","last_peered":"2023-10-18T16:07:41.806876+0000","last_clean":"2023-10-18T16:07:41.806876+0000","last_became_active":"2023-10-18T16:07:41.800562+0000","last_became_peered":"2023-10-18T16:07:41.800562+0000","last_unstale":"2023-10-18T16:07:41.806876+0000","last_undegraded":"2023-10-18T16:07:41.806876+0000","last_fullsized":"2023-10-18T16:07:41.806876+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.785692+0000","last_change":"2023-10-18T16:07:41.445361+0000","last_active":"2023-10-18T16:07:41.785692+0000","last_peered":"2023-10-18T16:07:41.785692+0000","last_clean":"2023-10-18T16:07:41.785692+0000","last_became_active":"2023-10-18T16:07:41.445159+0000","last_became_peered":"2023-10-18T16:07:41.445159+0000","last_unstale":"2023-10-18T16:07:41.785692+0000","last_undegraded":"2023-10-18T16:07:41.785692+0000","last_fullsized":"2023-10-18T16:07:41.785692+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.806606+0000","last_change":"2023-10-18T16:07:41.806046+0000","last_active":"2023-10-18T16:07:41.806606+0000","last_peered":"2023-10-18T16:07:41.806606+0000","last_clean":"2023-10-18T16:07:41.806606+0000","last_became_active":"2023-10-18T16:07:41.805799+0000","last_became_peered":"2023-10-18T16:07:41.805799+0000","last_unstale":"2023-10-18T16:07:41.806606+0000","last_undegraded":"2023-10-18T16:07:41.806606+0000","last_fullsized":"2023-10-18T16:07:41.806606+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.791286+0000","last_change":"2023-10-18T16:07:41.425946+0000","last_active":"2023-10-18T16:07:41.791286+0000","last_peered":"2023-10-18T16:07:41.791286+0000","last_clean":"2023-10-18T16:07:41.791286+0000","last_became_active":"2023-10-18T16:07:41.425673+0000","last_became_peered":"2023-10-18T16:07:41.425673+0000","last_unstale":"2023-10-18T16:07:41.791286+0000","last_undegraded":"2023-10-18T16:07:41.791286+0000","last_fullsized":"2023-10-18T16:07:41.791286+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.811135+0000","last_change":"2023-10-18T16:07:41.810501+0000","last_active":"2023-10-18T16:07:41.811135+0000","last_peered":"2023-10-18T16:07:41.811135+0000","last_clean":"2023-10-18T16:07:41.811135+0000","last_became_active":"2023-10-18T16:07:41.810233+0000","last_became_peered":"2023-10-18T16:07:41.810233+0000","last_unstale":"2023-10-18T16:07:41.811135+0000","last_undegraded":"2023-10-18T16:07:41.811135+0000","last_fullsized":"2023-10-18T16:07:41.811135+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.788717+0000","last_change":"2023-10-18T16:07:41.434399+0000","last_active":"2023-10-18T16:07:41.788717+0000","last_peered":"2023-10-18T16:07:41.788717+0000","last_clean":"2023-10-18T16:07:41.788717+0000","last_became_active":"2023-10-18T16:07:41.434143+0000","last_became_peered":"2023-10-18T16:07:41.434143+0000","last_unstale":"2023-10-18T16:07:41.788717+0000","last_undegraded":"2023-10-18T16:07:41.788717+0000","last_fullsized":"2023-10-18T16:07:41.788717+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.801283+0000","last_change":"2023-10-18T16:07:41.799082+0000","last_active":"2023-10-18T16:07:41.801283+0000","last_peered":"2023-10-18T16:07:41.801283+0000","last_clean":"2023-10-18T16:07:41.801283+0000","last_became_active":"2023-10-18T16:07:41.798807+0000","last_became_peered":"2023-10-18T16:07:41.798807+0000","last_unstale":"2023-10-18T16:07:41.801283+0000","last_undegraded":"2023-10-18T16:07:41.801283+0000","last_fullsized":"2023-10-18T16:07:41.801283+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.796407+0000","last_change":"2023-10-18T16:07:41.445579+0000","last_active":"2023-10-18T16:07:41.796407+0000","last_peered":"2023-10-18T16:07:41.796407+0000","last_clean":"2023-10-18T16:07:41.796407+0000","last_became_active":"2023-10-18T16:07:41.445348+0000","last_became_peered":"2023-10-18T16:07:41.445348+0000","last_unstale":"2023-10-18T16:07:41.796407+0000","last_undegraded":"2023-10-18T16:07:41.796407+0000","last_fullsized":"2023-10-18T16:07:41.796407+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.807820+0000","last_change":"2023-10-18T16:07:41.802347+0000","last_active":"2023-10-18T16:07:41.807820+0000","last_peered":"2023-10-18T16:07:41.807820+0000","last_clean":"2023-10-18T16:07:41.807820+0000","last_became_active":"2023-10-18T16:07:41.802061+0000","last_became_peered":"2023-10-18T16:07:41.802061+0000","last_unstale":"2023-10-18T16:07:41.807820+0000","last_undegraded":"2023-10-18T16:07:41.807820+0000","last_fullsized":"2023-10-18T16:07:41.807820+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.793332+0000","last_change":"2023-10-18T16:07:41.439414+0000","last_active":"2023-10-18T16:07:41.793332+0000","last_peered":"2023-10-18T16:07:41.793332+0000","last_clean":"2023-10-18T16:07:41.793332+0000","last_became_active":"2023-10-18T16:07:41.439137+0000","last_became_peered":"2023-10-18T16:07:41.439137+0000","last_unstale":"2023-10-18T16:07:41.793332+0000","last_undegraded":"2023-10-18T16:07:41.793332+0000","last_fullsized":"2023-10-18T16:07:41.793332+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.810429+0000","last_change":"2023-10-18T16:07:41.808426+0000","last_active":"2023-10-18T16:07:41.810429+0000","last_peered":"2023-10-18T16:07:41.810429+0000","last_clean":"2023-10-18T16:07:41.810429+0000","last_became_active":"2023-10-18T16:07:41.808146+0000","last_became_peered":"2023-10-18T16:07:41.808146+0000","last_unstale":"2023-10-18T16:07:41.810429+0000","last_undegraded":"2023-10-18T16:07:41.810429+0000","last_fullsized":"2023-10-18T16:07:41.810429+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.793112+0000","last_change":"2023-10-18T16:07:41.439810+0000","last_active":"2023-10-18T16:07:41.793112+0000","last_peered":"2023-10-18T16:07:41.793112+0000","last_clean":"2023-10-18T16:07:41.793112+0000","last_became_active":"2023-10-18T16:07:41.439545+0000","last_became_peered":"2023-10-18T16:07:41.439545+0000","last_unstale":"2023-10-18T16:07:41.793112+0000","last_undegraded":"2023-10-18T16:07:41.793112+0000","last_fullsized":"2023-10-18T16:07:41.793112+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.806045+0000","last_change":"2023-10-18T16:07:41.800617+0000","last_active":"2023-10-18T16:07:41.806045+0000","last_peered":"2023-10-18T16:07:41.806045+0000","last_clean":"2023-10-18T16:07:41.806045+0000","last_became_active":"2023-10-18T16:07:41.799998+0000","last_became_peered":"2023-10-18T16:07:41.799998+0000","last_unstale":"2023-10-18T16:07:41.806045+0000","last_undegraded":"2023-10-18T16:07:41.806045+0000","last_fullsized":"2023-10-18T16:07:41.806045+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.798396+0000","last_change":"2023-10-18T16:07:41.442796+0000","last_active":"2023-10-18T16:07:41.798396+0000","last_peered":"2023-10-18T16:07:41.798396+0000","last_clean":"2023-10-18T16:07:41.798396+0000","last_became_active":"2023-10-18T16:07:41.442007+0000","last_became_peered":"2023-10-18T16:07:41.442007+0000","last_unstale":"2023-10-18T16:07:41.798396+0000","last_undegraded":"2023-10-18T16:07:41.798396+0000","last_fullsized":"2023-10-18T16:07:41.798396+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3f","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.801310+0000","last_change":"2023-10-18T16:07:41.799046+0000","last_active":"2023-10-18T16:07:41.801310+0000","last_peered":"2023-10-18T16:07:41.801310+0000","last_clean":"2023-10-18T16:07:41.801310+0000","last_became_active":"2023-10-18T16:07:41.798754+0000","last_became_peered":"2023-10-18T16:07:41.798754+0000","last_unstale":"2023-10-18T16:07:41.801310+0000","last_undegraded":"2023-10-18T16:07:41.801310+0000","last_fullsized":"2023-10-18T16:07:41.801310+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.789919+0000","last_change":"2023-10-18T16:07:41.441443+0000","last_active":"2023-10-18T16:07:41.789919+0000","last_peered":"2023-10-18T16:07:41.789919+0000","last_clean":"2023-10-18T16:07:41.789919+0000","last_became_active":"2023-10-18T16:07:41.441208+0000","last_became_peered":"2023-10-18T16:07:41.441208+0000","last_unstale":"2023-10-18T16:07:41.789919+0000","last_undegraded":"2023-10-18T16:07:41.789919+0000","last_fullsized":"2023-10-18T16:07:41.789919+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.806457+0000","last_change":"2023-10-18T16:07:41.800460+0000","last_active":"2023-10-18T16:07:41.806457+0000","last_peered":"2023-10-18T16:07:41.806457+0000","last_clean":"2023-10-18T16:07:41.806457+0000","last_became_active":"2023-10-18T16:07:41.800013+0000","last_became_peered":"2023-10-18T16:07:41.800013+0000","last_unstale":"2023-10-18T16:07:41.806457+0000","last_undegraded":"2023-10-18T16:07:41.806457+0000","last_fullsized":"2023-10-18T16:07:41.806457+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.799786+0000","last_change":"2023-10-18T16:07:41.438711+0000","last_active":"2023-10-18T16:07:41.799786+0000","last_peered":"2023-10-18T16:07:41.799786+0000","last_clean":"2023-10-18T16:07:41.799786+0000","last_became_active":"2023-10-18T16:07:41.438408+0000","last_became_peered":"2023-10-18T16:07:41.438408+0000","last_unstale":"2023-10-18T16:07:41.799786+0000","last_undegraded":"2023-10-18T16:07:41.799786+0000","last_fullsized":"2023-10-18T16:07:41.799786+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.806161+0000","last_change":"2023-10-18T16:07:41.800718+0000","last_active":"2023-10-18T16:07:41.806161+0000","last_peered":"2023-10-18T16:07:41.806161+0000","last_clean":"2023-10-18T16:07:41.806161+0000","last_became_active":"2023-10-18T16:07:41.800400+0000","last_became_peered":"2023-10-18T16:07:41.800400+0000","last_unstale":"2023-10-18T16:07:41.806161+0000","last_undegraded":"2023-10-18T16:07:41.806161+0000","last_fullsized":"2023-10-18T16:07:41.806161+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.790414+0000","last_change":"2023-10-18T16:07:41.438151+0000","last_active":"2023-10-18T16:07:41.790414+0000","last_peered":"2023-10-18T16:07:41.790414+0000","last_clean":"2023-10-18T16:07:41.790414+0000","last_became_active":"2023-10-18T16:07:41.437504+0000","last_became_peered":"2023-10-18T16:07:41.437504+0000","last_unstale":"2023-10-18T16:07:41.790414+0000","last_undegraded":"2023-10-18T16:07:41.790414+0000","last_fullsized":"2023-10-18T16:07:41.790414+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.805013+0000","last_change":"2023-10-18T16:07:41.802411+0000","last_active":"2023-10-18T16:07:41.805013+0000","last_peered":"2023-10-18T16:07:41.805013+0000","last_clean":"2023-10-18T16:07:41.805013+0000","last_became_active":"2023-10-18T16:07:41.802124+0000","last_became_peered":"2023-10-18T16:07:41.802124+0000","last_unstale":"2023-10-18T16:07:41.805013+0000","last_undegraded":"2023-10-18T16:07:41.805013+0000","last_fullsized":"2023-10-18T16:07:41.805013+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.797104+0000","last_change":"2023-10-18T16:07:41.439071+0000","last_active":"2023-10-18T16:07:41.797104+0000","last_peered":"2023-10-18T16:07:41.797104+0000","last_clean":"2023-10-18T16:07:41.797104+0000","last_became_active":"2023-10-18T16:07:41.438144+0000","last_became_peered":"2023-10-18T16:07:41.438144+0000","last_unstale":"2023-10-18T16:07:41.797104+0000","last_undegraded":"2023-10-18T16:07:41.797104+0000","last_fullsized":"2023-10-18T16:07:41.797104+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.804818+0000","last_change":"2023-10-18T16:07:41.798403+0000","last_active":"2023-10-18T16:07:41.804818+0000","last_peered":"2023-10-18T16:07:41.804818+0000","last_clean":"2023-10-18T16:07:41.804818+0000","last_became_active":"2023-10-18T16:07:41.798149+0000","last_became_peered":"2023-10-18T16:07:41.798149+0000","last_unstale":"2023-10-18T16:07:41.804818+0000","last_undegraded":"2023-10-18T16:07:41.804818+0000","last_fullsized":"2023-10-18T16:07:41.804818+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.790253+0000","last_change":"2023-10-18T16:07:41.432914+0000","last_active":"2023-10-18T16:07:41.790253+0000","last_peered":"2023-10-18T16:07:41.790253+0000","last_clean":"2023-10-18T16:07:41.790253+0000","last_became_active":"2023-10-18T16:07:41.432601+0000","last_became_peered":"2023-10-18T16:07:41.432601+0000","last_unstale":"2023-10-18T16:07:41.790253+0000","last_undegraded":"2023-10-18T16:07:41.790253+0000","last_fullsized":"2023-10-18T16:07:41.790253+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.806049+0000","last_change":"2023-10-18T16:07:41.801443+0000","last_active":"2023-10-18T16:07:41.806049+0000","last_peered":"2023-10-18T16:07:41.806049+0000","last_clean":"2023-10-18T16:07:41.806049+0000","last_became_active":"2023-10-18T16:07:41.801271+0000","last_became_peered":"2023-10-18T16:07:41.801271+0000","last_unstale":"2023-10-18T16:07:41.806049+0000","last_undegraded":"2023-10-18T16:07:41.806049+0000","last_fullsized":"2023-10-18T16:07:41.806049+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.790644+0000","last_change":"2023-10-18T16:07:41.434525+0000","last_active":"2023-10-18T16:07:41.790644+0000","last_peered":"2023-10-18T16:07:41.790644+0000","last_clean":"2023-10-18T16:07:41.790644+0000","last_became_active":"2023-10-18T16:07:41.434220+0000","last_became_peered":"2023-10-18T16:07:41.434220+0000","last_unstale":"2023-10-18T16:07:41.790644+0000","last_undegraded":"2023-10-18T16:07:41.790644+0000","last_fullsized":"2023-10-18T16:07:41.790644+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.808917+0000","last_change":"2023-10-18T16:07:41.807740+0000","last_active":"2023-10-18T16:07:41.808917+0000","last_peered":"2023-10-18T16:07:41.808917+0000","last_clean":"2023-10-18T16:07:41.808917+0000","last_became_active":"2023-10-18T16:07:41.807473+0000","last_became_peered":"2023-10-18T16:07:41.807473+0000","last_unstale":"2023-10-18T16:07:41.808917+0000","last_undegraded":"2023-10-18T16:07:41.808917+0000","last_fullsized":"2023-10-18T16:07:41.808917+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.785962+0000","last_change":"2023-10-18T16:07:41.427326+0000","last_active":"2023-10-18T16:07:41.785962+0000","last_peered":"2023-10-18T16:07:41.785962+0000","last_clean":"2023-10-18T16:07:41.785962+0000","last_became_active":"2023-10-18T16:07:41.427083+0000","last_became_peered":"2023-10-18T16:07:41.427083+0000","last_unstale":"2023-10-18T16:07:41.785962+0000","last_undegraded":"2023-10-18T16:07:41.785962+0000","last_fullsized":"2023-10-18T16:07:41.785962+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.807852+0000","last_change":"2023-10-18T16:07:41.804453+0000","last_active":"2023-10-18T16:07:41.807852+0000","last_peered":"2023-10-18T16:07:41.807852+0000","last_clean":"2023-10-18T16:07:41.807852+0000","last_became_active":"2023-10-18T16:07:41.802577+0000","last_became_peered":"2023-10-18T16:07:41.802577+0000","last_unstale":"2023-10-18T16:07:41.807852+0000","last_undegraded":"2023-10-18T16:07:41.807852+0000","last_fullsized":"2023-10-18T16:07:41.807852+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:41.408605+0000","last_clean_scrub_stamp":"2023-10-18T16:07:41.408605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-18T16:07:41.798474+0000","last_change":"2023-10-18T16:07:41.436757+0000","last_active":"2023-10-18T16:07:41.798474+0000","last_peered":"2023-10-18T16:07:41.798474+0000","last_clean":"2023-10-18T16:07:41.798474+0000","last_became_active":"2023-10-18T16:07:41.436423+0000","last_became_peered":"2023-10-18T16:07:41.436423+0000","last_unstale":"2023-10-18T16:07:41.798474+0000","last_undegraded":"2023-10-18T16:07:41.798474+0000","last_fullsized":"2023-10-18T16:07:41.798474+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T16:07:40.406043+0000","last_clean_scrub_stamp":"2023-10-18T16:07:40.406043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297320,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074520,"statfs":{"total":96636764160,"available":96332308480,"internally_reserved":0,"allocated":229376,"data_stored":85341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56799999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67100000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297320,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074520,"statfs":{"total":96636764160,"available":96332308480,"internally_reserved":0,"allocated":229376,"data_stored":85341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.318}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48099999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]}]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297320,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074520,"statfs":{"total":96636764160,"available":96332308480,"internally_reserved":0,"allocated":229376,"data_stored":85341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26000000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33100000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.499}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65100000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297320,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074520,"statfs":{"total":96636764160,"available":96332308480,"internally_reserved":0,"allocated":229376,"data_stored":85341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34999999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36899999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60099999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297320,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074520,"statfs":{"total":96636764160,"available":96332308480,"internally_reserved":0,"allocated":229376,"data_stored":85341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42799999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.499}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71699999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68999999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297320,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074520,"statfs":{"total":96636764160,"available":96332308480,"internally_reserved":0,"allocated":229376,"data_stored":85341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34000000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.316}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52500000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59799999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53300000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61299999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297320,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074520,"statfs":{"total":96636764160,"available":96332308480,"internally_reserved":0,"allocated":229376,"data_stored":85341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40600000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.624}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55300000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64100000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297320,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074520,"statfs":{"total":96636764160,"available":96332308480,"internally_reserved":0,"allocated":229376,"data_stored":85341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51200000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67100000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60699999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66100000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.753}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-18T16:07:54.350 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-18T16:07:54.350 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-10-18T16:07:54.350 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-10-18T16:07:54.351 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:07:54.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.492+0000 7fe845d9d700 1 Processor -- start 2023-10-18T16:07:54.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.492+0000 7fe845d9d700 1 -- start start 2023-10-18T16:07:54.498 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.493+0000 7fe845d9d700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe8400b98a0 0x7fe8400b9c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:54.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.493+0000 7fe845d9d700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fe84014af00 0x7fe84014b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:54.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.493+0000 7fe845d9d700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe84014b880 con 0x7fe840067690 2023-10-18T16:07:54.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.493+0000 7fe845d9d700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8400661a0 con 0x7fe84014af00 2023-10-18T16:07:54.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.493+0000 7fe845d9d700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe840066440 con 0x7fe8400b98a0 2023-10-18T16:07:54.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.494+0000 7fe844d9b700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe8400b98a0 0x7fe8400b9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:54.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.494+0000 7fe84559c700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fe84014af00 0x7fe84014b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:54.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.494+0000 7fe844d9b700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe8400b98a0 0x7fe8400b9c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.88:54122/0 (socket says 172.21.15.88:54122) 2023-10-18T16:07:54.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.494+0000 7fe837fff700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fe840067690 0x7fe84014e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.88:52672/0 (socket says 172.21.15.88:52672) 2023-10-18T16:07:54.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.494+0000 7fe84559c700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fe84014af00 0x7fe84014b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.88:60328/0 (socket says 172.21.15.88:60328) 2023-10-18T16:07:54.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.494+0000 7fe844d9b700 1 -- 172.21.15.88:0/1905281736 learned_addr learned my addr 172.21.15.88:0/1905281736 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:54.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.494+0000 7fe84559c700 1 -- 172.21.15.88:0/1905281736 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe8400b98a0 msgr2=0x7fe8400b9c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:54.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.494+0000 7fe84559c700 1 --2- 172.21.15.88:0/1905281736 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe8400b98a0 0x7fe8400b9c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:54.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.494+0000 7fe84559c700 1 -- 172.21.15.88:0/1905281736 >> v1:172.21.15.151:6789/0 conn(0x7fe840067690 legacy=0x7fe84014e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:54.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.494+0000 7fe8377fe700 1 -- 172.21.15.88:0/1905281736 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2733877662 0 0) 0x7fe84014b880 con 0x7fe840067690 2023-10-18T16:07:54.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.494+0000 7fe84559c700 1 -- 172.21.15.88:0/1905281736 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe840066720 con 0x7fe84014af00 2023-10-18T16:07:54.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.494+0000 7fe844d9b700 1 --2- 172.21.15.88:0/1905281736 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe8400b98a0 0x7fe8400b9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2023-10-18T16:07:54.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.494+0000 7fe84559c700 1 --2- 172.21.15.88:0/1905281736 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe84014af00 0x7fe84014b390 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fe838009760 tx=0x7fe838009060).ready entity=mon.1 client_cookie=e74d35fcdef58fe1 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:54.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.494+0000 7fe8377fe700 1 -- 172.21.15.88:0/1905281736 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe83800b590 con 0x7fe84014af00 2023-10-18T16:07:54.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.494+0000 7fe8377fe700 1 -- 172.21.15.88:0/1905281736 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe83800b6f0 con 0x7fe84014af00 2023-10-18T16:07:54.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.494+0000 7fe8377fe700 1 -- 172.21.15.88:0/1905281736 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe83800ba70 con 0x7fe84014af00 2023-10-18T16:07:54.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.495+0000 7fe845d9d700 1 -- 172.21.15.88:0/1905281736 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe84014af00 msgr2=0x7fe84014b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:54.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.495+0000 7fe845d9d700 1 --2- 172.21.15.88:0/1905281736 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe84014af00 0x7fe84014b390 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fe838009760 tx=0x7fe838009060).stop 2023-10-18T16:07:54.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.495+0000 7fe845d9d700 1 -- 172.21.15.88:0/1905281736 shutdown_connections 2023-10-18T16:07:54.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.495+0000 7fe845d9d700 1 --2- 172.21.15.88:0/1905281736 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe84014af00 0x7fe84014b390 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:54.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.495+0000 7fe845d9d700 1 --2- 172.21.15.88:0/1905281736 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe8400b98a0 0x7fe8400b9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:54.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.495+0000 7fe845d9d700 1 -- 172.21.15.88:0/1905281736 >> 172.21.15.88:0/1905281736 conn(0x7fe840069120 msgr2=0x7fe840062650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:54.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.495+0000 7fe845d9d700 1 -- 172.21.15.88:0/1905281736 shutdown_connections 2023-10-18T16:07:54.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.495+0000 7fe845d9d700 1 -- 172.21.15.88:0/1905281736 wait complete. 2023-10-18T16:07:54.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.496+0000 7fe845d9d700 1 Processor -- start 2023-10-18T16:07:54.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.496+0000 7fe845d9d700 1 -- start start 2023-10-18T16:07:54.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.497+0000 7fe845d9d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe840067690 0x7fe8400be150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:54.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.497+0000 7fe845d9d700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe8400b98a0 0x7fe8400bbd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:54.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.497+0000 7fe845d9d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe84014af00 0x7fe8400c2630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:54.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.497+0000 7fe845d9d700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8400c4d90 con 0x7fe840067690 2023-10-18T16:07:54.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.497+0000 7fe845d9d700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8400be780 con 0x7fe84014af00 2023-10-18T16:07:54.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.497+0000 7fe845d9d700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8400c44f0 con 0x7fe8400b98a0 2023-10-18T16:07:54.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.497+0000 7fe844d9b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe840067690 0x7fe8400be150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:54.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.497+0000 7fe844d9b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe840067690 0x7fe8400be150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59782/0 (socket says 172.21.15.88:59782) 2023-10-18T16:07:54.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.497+0000 7fe844d9b700 1 -- 172.21.15.88:0/1055233950 learned_addr learned my addr 172.21.15.88:0/1055233950 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:54.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.497+0000 7fe84559c700 1 --2- 172.21.15.88:0/1055233950 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe84014af00 0x7fe8400c2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:54.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.497+0000 7fe837fff700 1 --2- 172.21.15.88:0/1055233950 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe8400b98a0 0x7fe8400bbd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:54.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.497+0000 7fe844d9b700 1 -- 172.21.15.88:0/1055233950 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe8400b98a0 msgr2=0x7fe8400bbd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:54.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.497+0000 7fe844d9b700 1 --2- 172.21.15.88:0/1055233950 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe8400b98a0 0x7fe8400bbd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:54.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.497+0000 7fe844d9b700 1 -- 172.21.15.88:0/1055233950 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe84014af00 msgr2=0x7fe8400c2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:54.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.497+0000 7fe844d9b700 1 --2- 172.21.15.88:0/1055233950 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe84014af00 0x7fe8400c2630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:54.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.497+0000 7fe844d9b700 1 -- 172.21.15.88:0/1055233950 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe83c009ae0 con 0x7fe840067690 2023-10-18T16:07:54.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.497+0000 7fe84559c700 1 --2- 172.21.15.88:0/1055233950 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe84014af00 0x7fe8400c2630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:07:54.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.497+0000 7fe844d9b700 1 --2- 172.21.15.88:0/1055233950 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe840067690 0x7fe8400be150 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fe83c00c170 tx=0x7fe83c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:54.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.498+0000 7fe834ff9700 1 -- 172.21.15.88:0/1055233950 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe83c00cc00 con 0x7fe840067690 2023-10-18T16:07:54.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.498+0000 7fe845d9d700 1 -- 172.21.15.88:0/1055233950 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8380092a0 con 0x7fe840067690 2023-10-18T16:07:54.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.498+0000 7fe834ff9700 1 -- 172.21.15.88:0/1055233950 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe83c00f040 con 0x7fe840067690 2023-10-18T16:07:54.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.498+0000 7fe834ff9700 1 -- 172.21.15.88:0/1055233950 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe83c014460 con 0x7fe840067690 2023-10-18T16:07:54.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.498+0000 7fe845d9d700 1 -- 172.21.15.88:0/1055233950 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8400c4ab0 con 0x7fe840067690 2023-10-18T16:07:54.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.499+0000 7fe845d9d700 1 -- 172.21.15.88:0/1055233950 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe840161110 con 0x7fe840067690 2023-10-18T16:07:54.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.499+0000 7fe834ff9700 1 -- 172.21.15.88:0/1055233950 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7fe83c014810 con 0x7fe840067690 2023-10-18T16:07:54.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.499+0000 7fe834ff9700 1 --2- 172.21.15.88:0/1055233950 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fe82c05f150 0x7fe82c061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:54.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.499+0000 7fe834ff9700 1 -- 172.21.15.88:0/1055233950 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6121+0+0 (secure 0 0 0) 0x7fe83c07a300 con 0x7fe840067690 2023-10-18T16:07:54.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.499+0000 7fe837fff700 1 --2- 172.21.15.88:0/1055233950 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fe82c05f150 0x7fe82c061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:54.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.500+0000 7fe837fff700 1 --2- 172.21.15.88:0/1055233950 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fe82c05f150 0x7fe82c061610 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fe8401485d0 tx=0x7fe830007180).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:54.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.503+0000 7fe834ff9700 1 -- 172.21.15.88:0/1055233950 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7fe83c04aac0 con 0x7fe840067690 2023-10-18T16:07:54.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.708+0000 7fe845d9d700 1 -- 172.21.15.88:0/1055233950 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe840069cd0 con 0x7fe840067690 2023-10-18T16:07:54.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.708+0000 7fe834ff9700 1 -- 172.21.15.88:0/1055233950 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7fe83c04b680 con 0x7fe840067690 2023-10-18T16:07:54.712 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:07:54.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.710+0000 7fe845d9d700 1 -- 172.21.15.88:0/1055233950 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fe82c05f150 msgr2=0x7fe82c061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:54.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.710+0000 7fe845d9d700 1 --2- 172.21.15.88:0/1055233950 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fe82c05f150 0x7fe82c061610 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fe8401485d0 tx=0x7fe830007180).stop 2023-10-18T16:07:54.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.710+0000 7fe845d9d700 1 -- 172.21.15.88:0/1055233950 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe840067690 msgr2=0x7fe8400be150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:54.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.710+0000 7fe845d9d700 1 --2- 172.21.15.88:0/1055233950 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe840067690 0x7fe8400be150 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fe83c00c170 tx=0x7fe83c00b040).stop 2023-10-18T16:07:54.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.710+0000 7fe845d9d700 1 -- 172.21.15.88:0/1055233950 shutdown_connections 2023-10-18T16:07:54.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.710+0000 7fe845d9d700 1 --2- 172.21.15.88:0/1055233950 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fe82c05f150 0x7fe82c061610 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:54.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.710+0000 7fe845d9d700 1 --2- 172.21.15.88:0/1055233950 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe84014af00 0x7fe8400c2630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:54.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.710+0000 7fe845d9d700 1 --2- 172.21.15.88:0/1055233950 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe840067690 0x7fe8400be150 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:54.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.710+0000 7fe845d9d700 1 --2- 172.21.15.88:0/1055233950 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe8400b98a0 0x7fe8400bbd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:54.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.710+0000 7fe845d9d700 1 -- 172.21.15.88:0/1055233950 >> 172.21.15.88:0/1055233950 conn(0x7fe840069120 msgr2=0x7fe840150ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:54.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.710+0000 7fe845d9d700 1 -- 172.21.15.88:0/1055233950 shutdown_connections 2023-10-18T16:07:54.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:54.710+0000 7fe845d9d700 1 -- 172.21.15.88:0/1055233950 wait complete. 2023-10-18T16:07:54.725 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:07:56.394 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~7s 2023-10-18T16:07:57.727 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:07:57.855 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.851+0000 7f58d72f2700 1 Processor -- start 2023-10-18T16:07:57.855 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.851+0000 7f58d72f2700 1 -- start start 2023-10-18T16:07:57.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.852+0000 7f58d72f2700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f58d01533f0 0x7f58d0151ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:57.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.852+0000 7f58d72f2700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58d00a9670 con 0x7f58d0153010 2023-10-18T16:07:57.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.852+0000 7f58d72f2700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58d0151a50 con 0x7f58d01523d0 2023-10-18T16:07:57.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.852+0000 7f58d72f2700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58d014df50 con 0x7f58d01533f0 2023-10-18T16:07:57.860 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.852+0000 7f58d508e700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f58d0153010 0x7f58d0064d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:47066/0 (socket says 172.21.15.88:47066) 2023-10-18T16:07:57.860 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.852+0000 7f58d508e700 1 -- 172.21.15.88:0/2401600172 learned_addr learned my addr 172.21.15.88:0/2401600172 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:07:57.860 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.852+0000 7f58c7fff700 1 -- 172.21.15.88:0/2401600172 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 392664528 0 0) 0x7f58d00a9670 con 0x7f58d0153010 2023-10-18T16:07:57.860 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.853+0000 7f58c7fff700 1 -- 172.21.15.88:0/2401600172 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58bc0033f0 con 0x7f58d0153010 2023-10-18T16:07:57.860 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.853+0000 7f58d488d700 1 --2- 172.21.15.88:0/2401600172 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f58d01533f0 0x7f58d0151ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:57.860 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.853+0000 7f58c7fff700 1 -- 172.21.15.88:0/2401600172 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2266602180 0 0) 0x7f58bc0033f0 con 0x7f58d0153010 2023-10-18T16:07:57.861 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.853+0000 7f58c7fff700 1 -- 172.21.15.88:0/2401600172 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f58d01533f0 msgr2=0x7f58d0151ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:57.861 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.853+0000 7f58c7fff700 1 --2- 172.21.15.88:0/2401600172 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f58d01533f0 0x7f58d0151ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:57.861 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.853+0000 7f58c7fff700 1 -- 172.21.15.88:0/2401600172 >> v1:172.21.15.151:6789/0 conn(0x7f58d01523d0 legacy=0x7f58d00c36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:57.861 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.853+0000 7f58c7fff700 1 -- 172.21.15.88:0/2401600172 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58d00a8af0 con 0x7f58d0153010 2023-10-18T16:07:57.861 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.853+0000 7f58c7fff700 1 -- 172.21.15.88:0/2401600172 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441715075 0 0) 0x7f58cc0017f0 con 0x7f58d0153010 2023-10-18T16:07:57.862 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.853+0000 7f58c7fff700 1 -- 172.21.15.88:0/2401600172 >> v1:172.21.15.88:6789/0 conn(0x7f58d0153010 legacy=0x7f58d0064d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:57.862 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.853+0000 7f58c7fff700 1 --2- 172.21.15.88:0/2401600172 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f58bc003910 0x7f58bc003d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:57.862 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.853+0000 7f58c7fff700 1 --2- 172.21.15.88:0/2401600172 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f58bc004690 0x7f58bc004b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:57.862 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.853+0000 7f58c7fff700 1 --2- 172.21.15.88:0/2401600172 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f58d01533f0 0x7f58bc0050b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:57.862 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.853+0000 7f58c7fff700 1 -- 172.21.15.88:0/2401600172 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58d014df50 con 0x7f58bc003910 2023-10-18T16:07:57.863 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.853+0000 7f58c7fff700 1 -- 172.21.15.88:0/2401600172 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58bc00b9c0 con 0x7f58bc004690 2023-10-18T16:07:57.863 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.853+0000 7f58c7fff700 1 -- 172.21.15.88:0/2401600172 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58bc00bb80 con 0x7f58d01533f0 2023-10-18T16:07:57.863 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.853+0000 7f58d508e700 1 --2- 172.21.15.88:0/2401600172 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f58bc003910 0x7f58bc003d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:57.863 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.853+0000 7f58d488d700 1 --2- 172.21.15.88:0/2401600172 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f58bc004690 0x7f58bc004b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:57.863 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.853+0000 7f58d588f700 1 --2- 172.21.15.88:0/2401600172 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f58d01533f0 0x7f58bc0050b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:57.863 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.853+0000 7f58d508e700 1 -- 172.21.15.88:0/2401600172 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f58d01533f0 msgr2=0x7f58bc0050b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:57.864 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.853+0000 7f58d508e700 1 --2- 172.21.15.88:0/2401600172 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f58d01533f0 0x7f58bc0050b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:57.864 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.853+0000 7f58d508e700 1 -- 172.21.15.88:0/2401600172 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f58bc004690 msgr2=0x7f58bc004b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:07:57.864 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.853+0000 7f58d508e700 1 --2- 172.21.15.88:0/2401600172 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f58bc004690 0x7f58bc004b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:57.864 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.853+0000 7f58d508e700 1 -- 172.21.15.88:0/2401600172 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58bc0033f0 con 0x7f58bc003910 2023-10-18T16:07:57.864 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.853+0000 7f58d488d700 1 --2- 172.21.15.88:0/2401600172 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f58bc004690 0x7f58bc004b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:07:57.864 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.853+0000 7f58d508e700 1 --2- 172.21.15.88:0/2401600172 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f58bc003910 0x7f58bc003d30 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f58cc00c480 tx=0x7f58cc00c790).ready entity=mon.0 client_cookie=5ad5ac3098e23cf0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:57.865 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.853+0000 7f58c7fff700 1 -- 172.21.15.88:0/2401600172 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58cc002030 con 0x7f58bc003910 2023-10-18T16:07:57.865 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.853+0000 7f58c7fff700 1 -- 172.21.15.88:0/2401600172 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58cc00ce20 con 0x7f58bc003910 2023-10-18T16:07:57.865 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.853+0000 7f58c7fff700 1 -- 172.21.15.88:0/2401600172 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58cc005d30 con 0x7f58bc003910 2023-10-18T16:07:57.865 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.854+0000 7f58d72f2700 1 -- 172.21.15.88:0/2401600172 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f58bc003910 msgr2=0x7f58bc003d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:57.865 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.854+0000 7f58d72f2700 1 --2- 172.21.15.88:0/2401600172 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f58bc003910 0x7f58bc003d30 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f58cc00c480 tx=0x7f58cc00c790).stop 2023-10-18T16:07:57.866 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.854+0000 7f58d588f700 1 -- 172.21.15.88:0/2401600172 reap_dead start 2023-10-18T16:07:57.866 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.854+0000 7f58d72f2700 1 -- 172.21.15.88:0/2401600172 shutdown_connections 2023-10-18T16:07:57.866 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.854+0000 7f58d72f2700 1 -- 172.21.15.88:0/2401600172 >> 172.21.15.88:0/2401600172 conn(0x7f58d0069d80 msgr2=0x7f58d006a190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:57.866 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.854+0000 7f58d72f2700 1 -- 172.21.15.88:0/2401600172 shutdown_connections 2023-10-18T16:07:57.866 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.854+0000 7f58d72f2700 1 -- 172.21.15.88:0/2401600172 wait complete. 2023-10-18T16:07:57.866 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.855+0000 7f58d72f2700 1 Processor -- start 2023-10-18T16:07:57.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.855+0000 7f58d72f2700 1 -- start start 2023-10-18T16:07:57.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.856+0000 7f58d72f2700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f58bc003910 0x7f58d0160a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:57.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.856+0000 7f58d72f2700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f58d00c7210 0x7f58d00bcdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:57.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.856+0000 7f58d72f2700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f58d00bd2b0 0x7f58d00bb8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:57.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.856+0000 7f58d72f2700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58d0160f60 con 0x7f58d00bd2b0 2023-10-18T16:07:57.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.856+0000 7f58d72f2700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58d0150b10 con 0x7f58bc003910 2023-10-18T16:07:57.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.856+0000 7f58d72f2700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58d00bd790 con 0x7f58d00c7210 2023-10-18T16:07:57.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.856+0000 7f58d588f700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f58d00bd2b0 0x7f58d00bb8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:57.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.856+0000 7f58d588f700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f58d00bd2b0 0x7f58d00bb8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38464/0 (socket says 172.21.15.88:38464) 2023-10-18T16:07:57.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.856+0000 7f58d588f700 1 -- 172.21.15.88:0/888258051 learned_addr learned my addr 172.21.15.88:0/888258051 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:07:57.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.856+0000 7f58d508e700 1 --2- 172.21.15.88:0/888258051 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f58bc003910 0x7f58d0160a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:57.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.856+0000 7f58d488d700 1 --2- 172.21.15.88:0/888258051 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f58d00c7210 0x7f58d00bcdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:57.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.856+0000 7f58d588f700 1 -- 172.21.15.88:0/888258051 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f58d00c7210 msgr2=0x7f58d00bcdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:57.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.856+0000 7f58d588f700 1 --2- 172.21.15.88:0/888258051 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f58d00c7210 0x7f58d00bcdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:57.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.856+0000 7f58d588f700 1 -- 172.21.15.88:0/888258051 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f58bc003910 msgr2=0x7f58d0160a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:57.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.856+0000 7f58d588f700 1 --2- 172.21.15.88:0/888258051 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f58bc003910 0x7f58d0160a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:57.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.856+0000 7f58d588f700 1 -- 172.21.15.88:0/888258051 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58d014ff00 con 0x7f58d00bd2b0 2023-10-18T16:07:57.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.857+0000 7f58d588f700 1 --2- 172.21.15.88:0/888258051 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f58d00bd2b0 0x7f58d00bb8d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f58d00840b0 tx=0x7f58c8009040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:57.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.857+0000 7f58c57fa700 1 -- 172.21.15.88:0/888258051 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58c8014430 con 0x7f58d00bd2b0 2023-10-18T16:07:57.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.857+0000 7f58c57fa700 1 -- 172.21.15.88:0/888258051 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58c8014590 con 0x7f58d00bd2b0 2023-10-18T16:07:57.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.857+0000 7f58c57fa700 1 -- 172.21.15.88:0/888258051 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58c8014910 con 0x7f58d00bd2b0 2023-10-18T16:07:57.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.857+0000 7f58d72f2700 1 -- 172.21.15.88:0/888258051 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58d01501e0 con 0x7f58d00bd2b0 2023-10-18T16:07:57.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.857+0000 7f58d72f2700 1 -- 172.21.15.88:0/888258051 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58d01507a0 con 0x7f58d00bd2b0 2023-10-18T16:07:57.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.858+0000 7f58d72f2700 1 -- 172.21.15.88:0/888258051 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58d0150cc0 con 0x7f58d00bd2b0 2023-10-18T16:07:57.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.858+0000 7f58c57fa700 1 -- 172.21.15.88:0/888258051 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f58c8014d00 con 0x7f58d00bd2b0 2023-10-18T16:07:57.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.858+0000 7f58c57fa700 1 --2- 172.21.15.88:0/888258051 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f58b405f120 0x7f58b40615e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:07:57.874 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.858+0000 7f58c57fa700 1 -- 172.21.15.88:0/888258051 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6121+0+0 (secure 0 0 0) 0x7f58c8079a10 con 0x7f58d00bd2b0 2023-10-18T16:07:57.874 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.859+0000 7f58d508e700 1 --2- 172.21.15.88:0/888258051 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f58b405f120 0x7f58b40615e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:07:57.874 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.859+0000 7f58d508e700 1 --2- 172.21.15.88:0/888258051 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f58b405f120 0x7f58b40615e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f58cc00c7d0 tx=0x7f58cc015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:07:57.874 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:57.862+0000 7f58c57fa700 1 -- 172.21.15.88:0/888258051 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f58c8049ac0 con 0x7f58d00bd2b0 2023-10-18T16:07:58.063 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:58.060+0000 7f58d72f2700 1 -- 172.21.15.88:0/888258051 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f58d00654a0 con 0x7f58d00bd2b0 2023-10-18T16:07:58.064 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:58.060+0000 7f58c57fa700 1 -- 172.21.15.88:0/888258051 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7f58d0150cc0 con 0x7f58d00bd2b0 2023-10-18T16:07:58.064 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:07:58.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:58.062+0000 7f58d72f2700 1 -- 172.21.15.88:0/888258051 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f58b405f120 msgr2=0x7f58b40615e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:58.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:58.062+0000 7f58d72f2700 1 --2- 172.21.15.88:0/888258051 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f58b405f120 0x7f58b40615e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f58cc00c7d0 tx=0x7f58cc015000).stop 2023-10-18T16:07:58.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:58.062+0000 7f58d72f2700 1 -- 172.21.15.88:0/888258051 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f58d00bd2b0 msgr2=0x7f58d00bb8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:07:58.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:58.062+0000 7f58d72f2700 1 --2- 172.21.15.88:0/888258051 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f58d00bd2b0 0x7f58d00bb8d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f58d00840b0 tx=0x7f58c8009040).stop 2023-10-18T16:07:58.067 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:58.062+0000 7f58d72f2700 1 -- 172.21.15.88:0/888258051 shutdown_connections 2023-10-18T16:07:58.067 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:58.062+0000 7f58d72f2700 1 --2- 172.21.15.88:0/888258051 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f58b405f120 0x7f58b40615e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:58.067 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:58.062+0000 7f58d72f2700 1 --2- 172.21.15.88:0/888258051 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f58d00bd2b0 0x7f58d00bb8d0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:58.067 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:58.062+0000 7f58d72f2700 1 --2- 172.21.15.88:0/888258051 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f58bc003910 0x7f58d0160a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:58.068 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:58.062+0000 7f58d72f2700 1 --2- 172.21.15.88:0/888258051 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f58d00c7210 0x7f58d00bcdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:07:58.068 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:58.062+0000 7f58d72f2700 1 -- 172.21.15.88:0/888258051 >> 172.21.15.88:0/888258051 conn(0x7f58d0069d80 msgr2=0x7f58d014cf90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:07:58.068 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:58.062+0000 7f58d72f2700 1 -- 172.21.15.88:0/888258051 shutdown_connections 2023-10-18T16:07:58.068 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:07:58.062+0000 7f58d72f2700 1 -- 172.21.15.88:0/888258051 wait complete. 2023-10-18T16:07:58.077 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:08:01.078 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:08:01.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.199+0000 7fe682991700 1 Processor -- start 2023-10-18T16:08:01.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.199+0000 7fe682991700 1 -- start start 2023-10-18T16:08:01.206 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.200+0000 7fe682991700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe67c0bd8f0 0x7fe67c0bed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:01.206 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.200+0000 7fe682991700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe67c0be160 0x7fe67c0be5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:01.206 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.200+0000 7fe682991700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe67c0bafb0 con 0x7fe67c1511c0 2023-10-18T16:08:01.207 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.200+0000 7fe682991700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe67c0bdcd0 con 0x7fe67c0bd8f0 2023-10-18T16:08:01.207 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.200+0000 7fe682991700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe67c0beab0 con 0x7fe67c0be160 2023-10-18T16:08:01.207 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.200+0000 7fe67bfff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe67c0bd8f0 0x7fe67c0bed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:01.207 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.200+0000 7fe67bfff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe67c0bd8f0 0x7fe67c0bed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38466/0 (socket says 172.21.15.88:38466) 2023-10-18T16:08:01.207 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.200+0000 7fe67bfff700 1 -- 172.21.15.88:0/3319806408 learned_addr learned my addr 172.21.15.88:0/3319806408 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:08:01.208 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.200+0000 7fe67b7fe700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fe67c1511c0 0x7fe67c0bc4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:47082/0 (socket says 172.21.15.88:47082) 2023-10-18T16:08:01.208 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.201+0000 7fe67bfff700 1 -- 172.21.15.88:0/3319806408 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe67c0be160 msgr2=0x7fe67c0be5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:08:01.208 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.201+0000 7fe67bfff700 1 --2- 172.21.15.88:0/3319806408 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe67c0be160 0x7fe67c0be5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:08:01.208 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.201+0000 7fe67bfff700 1 -- 172.21.15.88:0/3319806408 >> v1:172.21.15.88:6789/0 conn(0x7fe67c1511c0 legacy=0x7fe67c0bc4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:01.208 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.201+0000 7fe67bfff700 1 -- 172.21.15.88:0/3319806408 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe67c067d60 con 0x7fe67c0bd8f0 2023-10-18T16:08:01.209 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.201+0000 7fe67affd700 1 -- 172.21.15.88:0/3319806408 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3610574764 0 0) 0x7fe67c0bafb0 con 0x7fe67c1511c0 2023-10-18T16:08:01.209 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.201+0000 7fe67bfff700 1 --2- 172.21.15.88:0/3319806408 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe67c0bd8f0 0x7fe67c0bed70 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fe66c006fd0 tx=0x7fe66c0090f0).ready entity=mon.0 client_cookie=a021cede96a45943 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:08:01.209 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.201+0000 7fe67affd700 1 -- 172.21.15.88:0/3319806408 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe66c00b690 con 0x7fe67c0bd8f0 2023-10-18T16:08:01.209 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.201+0000 7fe67affd700 1 -- 172.21.15.88:0/3319806408 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe66c00b7f0 con 0x7fe67c0bd8f0 2023-10-18T16:08:01.209 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.201+0000 7fe67affd700 1 -- 172.21.15.88:0/3319806408 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe66c00bb70 con 0x7fe67c0bd8f0 2023-10-18T16:08:01.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.201+0000 7fe682991700 1 -- 172.21.15.88:0/3319806408 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe67c0bd8f0 msgr2=0x7fe67c0bed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:01.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.201+0000 7fe682991700 1 --2- 172.21.15.88:0/3319806408 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe67c0bd8f0 0x7fe67c0bed70 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fe66c006fd0 tx=0x7fe66c0090f0).stop 2023-10-18T16:08:01.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.201+0000 7fe682991700 1 -- 172.21.15.88:0/3319806408 shutdown_connections 2023-10-18T16:08:01.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.201+0000 7fe682991700 1 --2- 172.21.15.88:0/3319806408 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe67c0be160 0x7fe67c0be5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:08:01.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.201+0000 7fe682991700 1 --2- 172.21.15.88:0/3319806408 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe67c0bd8f0 0x7fe67c0bed70 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:01.211 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.201+0000 7fe682991700 1 -- 172.21.15.88:0/3319806408 >> 172.21.15.88:0/3319806408 conn(0x7fe67c063760 msgr2=0x7fe67c063b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:08:01.211 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.202+0000 7fe682991700 1 -- 172.21.15.88:0/3319806408 shutdown_connections 2023-10-18T16:08:01.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.202+0000 7fe682991700 1 -- 172.21.15.88:0/3319806408 wait complete. 2023-10-18T16:08:01.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.203+0000 7fe682991700 1 Processor -- start 2023-10-18T16:08:01.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.203+0000 7fe682991700 1 -- start start 2023-10-18T16:08:01.216 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.203+0000 7fe682991700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe67c0bd8f0 0x7fe67c1508c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:01.216 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.203+0000 7fe682991700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe67c0be160 0x7fe67c15d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:01.216 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.203+0000 7fe682991700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe67c1511c0 0x7fe67c15db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:01.216 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.204+0000 7fe682991700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe67c1603d0 con 0x7fe67c1511c0 2023-10-18T16:08:01.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.204+0000 7fe682991700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe67c15e020 con 0x7fe67c0bd8f0 2023-10-18T16:08:01.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.204+0000 7fe682991700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe67c149990 con 0x7fe67c0be160 2023-10-18T16:08:01.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.204+0000 7fe680f2e700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe67c1511c0 0x7fe67c15db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:01.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.204+0000 7fe680f2e700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe67c1511c0 0x7fe67c15db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38468/0 (socket says 172.21.15.88:38468) 2023-10-18T16:08:01.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.204+0000 7fe680f2e700 1 -- 172.21.15.88:0/449040810 learned_addr learned my addr 172.21.15.88:0/449040810 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:08:01.218 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.204+0000 7fe67b7fe700 1 --2- 172.21.15.88:0/449040810 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe67c0be160 0x7fe67c15d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:01.218 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.204+0000 7fe67bfff700 1 --2- 172.21.15.88:0/449040810 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe67c0bd8f0 0x7fe67c1508c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:01.218 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.204+0000 7fe680f2e700 1 -- 172.21.15.88:0/449040810 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe67c0be160 msgr2=0x7fe67c15d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:01.218 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.204+0000 7fe680f2e700 1 --2- 172.21.15.88:0/449040810 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe67c0be160 0x7fe67c15d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:01.218 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.204+0000 7fe680f2e700 1 -- 172.21.15.88:0/449040810 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe67c0bd8f0 msgr2=0x7fe67c1508c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:01.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.204+0000 7fe680f2e700 1 --2- 172.21.15.88:0/449040810 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe67c0bd8f0 0x7fe67c1508c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:01.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.204+0000 7fe680f2e700 1 -- 172.21.15.88:0/449040810 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe66c009370 con 0x7fe67c1511c0 2023-10-18T16:08:01.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.204+0000 7fe67b7fe700 1 --2- 172.21.15.88:0/449040810 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe67c0be160 0x7fe67c15d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:08:01.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.204+0000 7fe67bfff700 1 --2- 172.21.15.88:0/449040810 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe67c0bd8f0 0x7fe67c1508c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:08:01.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.204+0000 7fe680f2e700 1 --2- 172.21.15.88:0/449040810 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe67c1511c0 0x7fe67c15db30 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7fe67400bc30 tx=0x7fe674009840).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:08:01.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.204+0000 7fe663fff700 1 -- 172.21.15.88:0/449040810 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe67400c7a0 con 0x7fe67c1511c0 2023-10-18T16:08:01.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.204+0000 7fe663fff700 1 -- 172.21.15.88:0/449040810 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe674010070 con 0x7fe67c1511c0 2023-10-18T16:08:01.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.204+0000 7fe682991700 1 -- 172.21.15.88:0/449040810 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe67c15f750 con 0x7fe67c1511c0 2023-10-18T16:08:01.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.204+0000 7fe663fff700 1 -- 172.21.15.88:0/449040810 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe67400c960 con 0x7fe67c1511c0 2023-10-18T16:08:01.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.204+0000 7fe682991700 1 -- 172.21.15.88:0/449040810 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe67c15fd10 con 0x7fe67c1511c0 2023-10-18T16:08:01.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.205+0000 7fe682991700 1 -- 172.21.15.88:0/449040810 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe67c160510 con 0x7fe67c1511c0 2023-10-18T16:08:01.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.205+0000 7fe663fff700 1 -- 172.21.15.88:0/449040810 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7fe67400cd10 con 0x7fe67c1511c0 2023-10-18T16:08:01.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.206+0000 7fe663fff700 1 --2- 172.21.15.88:0/449040810 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fe66805f150 0x7fe668061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:01.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.206+0000 7fe663fff700 1 -- 172.21.15.88:0/449040810 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6121+0+0 (secure 0 0 0) 0x7fe674079ae0 con 0x7fe67c1511c0 2023-10-18T16:08:01.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.206+0000 7fe67bfff700 1 --2- 172.21.15.88:0/449040810 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fe66805f150 0x7fe668061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:01.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.206+0000 7fe67bfff700 1 --2- 172.21.15.88:0/449040810 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fe66805f150 0x7fe668061610 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7fe66c009130 tx=0x7fe66c017000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:08:01.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.210+0000 7fe663fff700 1 -- 172.21.15.88:0/449040810 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7fe67c160510 con 0x7fe67c1511c0 2023-10-18T16:08:01.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.410+0000 7fe682991700 1 -- 172.21.15.88:0/449040810 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe67c160510 con 0x7fe67c1511c0 2023-10-18T16:08:01.414 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.410+0000 7fe663fff700 1 -- 172.21.15.88:0/449040810 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7fe67c160510 con 0x7fe67c1511c0 2023-10-18T16:08:01.415 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:08:01.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.412+0000 7fe682991700 1 -- 172.21.15.88:0/449040810 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fe66805f150 msgr2=0x7fe668061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:01.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.412+0000 7fe682991700 1 --2- 172.21.15.88:0/449040810 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fe66805f150 0x7fe668061610 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7fe66c009130 tx=0x7fe66c017000).stop 2023-10-18T16:08:01.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.412+0000 7fe682991700 1 -- 172.21.15.88:0/449040810 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe67c1511c0 msgr2=0x7fe67c15db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:01.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.412+0000 7fe682991700 1 --2- 172.21.15.88:0/449040810 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe67c1511c0 0x7fe67c15db30 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7fe67400bc30 tx=0x7fe674009840).stop 2023-10-18T16:08:01.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.412+0000 7fe682991700 1 -- 172.21.15.88:0/449040810 shutdown_connections 2023-10-18T16:08:01.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.412+0000 7fe682991700 1 --2- 172.21.15.88:0/449040810 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fe66805f150 0x7fe668061610 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:01.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.412+0000 7fe682991700 1 --2- 172.21.15.88:0/449040810 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe67c1511c0 0x7fe67c15db30 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:01.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.412+0000 7fe682991700 1 --2- 172.21.15.88:0/449040810 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe67c0bd8f0 0x7fe67c1508c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:01.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.412+0000 7fe682991700 1 --2- 172.21.15.88:0/449040810 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe67c0be160 0x7fe67c15d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:01.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.412+0000 7fe682991700 1 -- 172.21.15.88:0/449040810 >> 172.21.15.88:0/449040810 conn(0x7fe67c063760 msgr2=0x7fe67c0bcde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:08:01.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.412+0000 7fe682991700 1 -- 172.21.15.88:0/449040810 shutdown_connections 2023-10-18T16:08:01.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:01.412+0000 7fe682991700 1 -- 172.21.15.88:0/449040810 wait complete. 2023-10-18T16:08:01.427 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:08:03.207 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~14s 2023-10-18T16:08:04.428 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:08:04.554 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.550+0000 7f88dde67700 1 Processor -- start 2023-10-18T16:08:04.554 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.550+0000 7f88dde67700 1 -- start start 2023-10-18T16:08:04.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.551+0000 7f88dde67700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f88d80c6a60 0x7f88d80c6e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:04.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.551+0000 7f88dde67700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f88d8063d70 0x7f88d8153660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:04.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.551+0000 7f88dde67700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88d8151020 con 0x7f88d8153b50 2023-10-18T16:08:04.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.551+0000 7f88dde67700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88d8150090 con 0x7f88d8063d70 2023-10-18T16:08:04.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.551+0000 7f88dde67700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88d8150b90 con 0x7f88d80c6a60 2023-10-18T16:08:04.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.551+0000 7f88cffff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f88d8063d70 0x7f88d8153660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:04.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.551+0000 7f88cffff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f88d8063d70 0x7f88d8153660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38482/0 (socket says 172.21.15.88:38482) 2023-10-18T16:08:04.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.551+0000 7f88cffff700 1 -- 172.21.15.88:0/1538053079 learned_addr learned my addr 172.21.15.88:0/1538053079 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:08:04.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.551+0000 7f88dce65700 1 --2- 172.21.15.88:0/1538053079 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f88d80c6a60 0x7f88d80c6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:04.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.551+0000 7f88cffff700 1 -- 172.21.15.88:0/1538053079 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f88d80c6a60 msgr2=0x7f88d80c6e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:08:04.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.551+0000 7f88cffff700 1 --2- 172.21.15.88:0/1538053079 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f88d80c6a60 0x7f88d80c6e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:04.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.551+0000 7f88cffff700 1 -- 172.21.15.88:0/1538053079 >> v1:172.21.15.151:6789/0 conn(0x7f88d8153b50 legacy=0x7f88d80c2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:04.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.551+0000 7f88cffff700 1 -- 172.21.15.88:0/1538053079 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88d80c2f70 con 0x7f88d8063d70 2023-10-18T16:08:04.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.551+0000 7f88dce65700 1 --2- 172.21.15.88:0/1538053079 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f88d80c6a60 0x7f88d80c6e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-10-18T16:08:04.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.552+0000 7f88cffff700 1 --2- 172.21.15.88:0/1538053079 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f88d8063d70 0x7f88d8153660 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f88c8006fd0 tx=0x7f88c80090f0).ready entity=mon.0 client_cookie=5c459efe4c7d9650 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:08:04.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.552+0000 7f88cf7fe700 1 -- 172.21.15.88:0/1538053079 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c800b600 con 0x7f88d8063d70 2023-10-18T16:08:04.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.552+0000 7f88cf7fe700 1 -- 172.21.15.88:0/1538053079 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88c800b760 con 0x7f88d8063d70 2023-10-18T16:08:04.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.552+0000 7f88cf7fe700 1 -- 172.21.15.88:0/1538053079 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c800bae0 con 0x7f88d8063d70 2023-10-18T16:08:04.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.552+0000 7f88dde67700 1 -- 172.21.15.88:0/1538053079 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f88d8063d70 msgr2=0x7f88d8153660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:04.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.552+0000 7f88dde67700 1 --2- 172.21.15.88:0/1538053079 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f88d8063d70 0x7f88d8153660 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f88c8006fd0 tx=0x7f88c80090f0).stop 2023-10-18T16:08:04.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.552+0000 7f88dde67700 1 -- 172.21.15.88:0/1538053079 shutdown_connections 2023-10-18T16:08:04.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.552+0000 7f88dde67700 1 --2- 172.21.15.88:0/1538053079 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f88d80c6a60 0x7f88d80c6e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:04.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.552+0000 7f88dde67700 1 --2- 172.21.15.88:0/1538053079 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f88d8063d70 0x7f88d8153660 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:04.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.552+0000 7f88dde67700 1 -- 172.21.15.88:0/1538053079 >> 172.21.15.88:0/1538053079 conn(0x7f88d805e050 msgr2=0x7f88d805e460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:08:04.562 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.552+0000 7f88dde67700 1 -- 172.21.15.88:0/1538053079 shutdown_connections 2023-10-18T16:08:04.562 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.552+0000 7f88dde67700 1 -- 172.21.15.88:0/1538053079 wait complete. 2023-10-18T16:08:04.562 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.553+0000 7f88dde67700 1 Processor -- start 2023-10-18T16:08:04.562 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.554+0000 7f88dde67700 1 -- start start 2023-10-18T16:08:04.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.554+0000 7f88dde67700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f88d8063d70 0x7f88d80be520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:04.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.554+0000 7f88dde67700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f88d80c6a60 0x7f88d80c4f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:04.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.554+0000 7f88dde67700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f88d8153b50 0x7f88d80c5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:04.566 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.554+0000 7f88dde67700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88d8151ea0 con 0x7f88d80c6a60 2023-10-18T16:08:04.566 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.554+0000 7f88dde67700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88d81478b0 con 0x7f88d8153b50 2023-10-18T16:08:04.566 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.554+0000 7f88dde67700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88d8147b70 con 0x7f88d8063d70 2023-10-18T16:08:04.566 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.554+0000 7f88cffff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f88d80c6a60 0x7f88d80c4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:04.567 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.554+0000 7f88cffff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f88d80c6a60 0x7f88d80c4f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38492/0 (socket says 172.21.15.88:38492) 2023-10-18T16:08:04.567 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.554+0000 7f88cffff700 1 -- 172.21.15.88:0/3816563702 learned_addr learned my addr 172.21.15.88:0/3816563702 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:08:04.567 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.555+0000 7f88dce65700 1 --2- 172.21.15.88:0/3816563702 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f88d8063d70 0x7f88d80be520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:04.567 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.555+0000 7f88dd666700 1 --2- 172.21.15.88:0/3816563702 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f88d8153b50 0x7f88d80c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:04.567 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.555+0000 7f88cffff700 1 -- 172.21.15.88:0/3816563702 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f88d8063d70 msgr2=0x7f88d80be520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:04.567 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.555+0000 7f88cffff700 1 --2- 172.21.15.88:0/3816563702 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f88d8063d70 0x7f88d80be520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:04.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.555+0000 7f88cffff700 1 -- 172.21.15.88:0/3816563702 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f88d8153b50 msgr2=0x7f88d80c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:04.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.555+0000 7f88cffff700 1 --2- 172.21.15.88:0/3816563702 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f88d8153b50 0x7f88d80c5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:04.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.555+0000 7f88cffff700 1 -- 172.21.15.88:0/3816563702 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88c8009370 con 0x7f88d80c6a60 2023-10-18T16:08:04.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.555+0000 7f88cffff700 1 --2- 172.21.15.88:0/3816563702 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f88d80c6a60 0x7f88d80c4f20 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f88d805ec40 tx=0x7f88c8017000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:08:04.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.555+0000 7f88ccff9700 1 -- 172.21.15.88:0/3816563702 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c800bae0 con 0x7f88d80c6a60 2023-10-18T16:08:04.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.555+0000 7f88ccff9700 1 -- 172.21.15.88:0/3816563702 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88c8000c90 con 0x7f88d80c6a60 2023-10-18T16:08:04.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.555+0000 7f88dde67700 1 -- 172.21.15.88:0/3816563702 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88d80bea10 con 0x7f88d80c6a60 2023-10-18T16:08:04.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.555+0000 7f88ccff9700 1 -- 172.21.15.88:0/3816563702 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c801c670 con 0x7f88d80c6a60 2023-10-18T16:08:04.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.555+0000 7f88dde67700 1 -- 172.21.15.88:0/3816563702 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88d81520b0 con 0x7f88d80c6a60 2023-10-18T16:08:04.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.556+0000 7f88dde67700 1 -- 172.21.15.88:0/3816563702 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88d8084340 con 0x7f88d80c6a60 2023-10-18T16:08:04.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.556+0000 7f88ccff9700 1 -- 172.21.15.88:0/3816563702 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f88c801ca20 con 0x7f88d80c6a60 2023-10-18T16:08:04.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.557+0000 7f88ccff9700 1 --2- 172.21.15.88:0/3816563702 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f88c405f150 0x7f88c4061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:04.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.557+0000 7f88ccff9700 1 -- 172.21.15.88:0/3816563702 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6121+0+0 (secure 0 0 0) 0x7f88c807bf10 con 0x7f88d80c6a60 2023-10-18T16:08:04.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.557+0000 7f88dce65700 1 --2- 172.21.15.88:0/3816563702 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f88c405f150 0x7f88c4061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:04.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.557+0000 7f88dce65700 1 --2- 172.21.15.88:0/3816563702 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f88c405f150 0x7f88c4061610 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f88d400aa70 tx=0x7f88d4008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:08:04.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.561+0000 7f88ccff9700 1 -- 172.21.15.88:0/3816563702 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f88c804d1e0 con 0x7f88d80c6a60 2023-10-18T16:08:04.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.760+0000 7f88dde67700 1 -- 172.21.15.88:0/3816563702 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f88d8046c20 con 0x7f88d80c6a60 2023-10-18T16:08:04.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.761+0000 7f88ccff9700 1 -- 172.21.15.88:0/3816563702 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7f88c8050800 con 0x7f88d80c6a60 2023-10-18T16:08:04.765 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:08:04.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.762+0000 7f88dde67700 1 -- 172.21.15.88:0/3816563702 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f88c405f150 msgr2=0x7f88c4061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:04.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.762+0000 7f88dde67700 1 --2- 172.21.15.88:0/3816563702 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f88c405f150 0x7f88c4061610 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f88d400aa70 tx=0x7f88d4008040).stop 2023-10-18T16:08:04.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.762+0000 7f88dde67700 1 -- 172.21.15.88:0/3816563702 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f88d80c6a60 msgr2=0x7f88d80c4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:04.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.762+0000 7f88dde67700 1 --2- 172.21.15.88:0/3816563702 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f88d80c6a60 0x7f88d80c4f20 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f88d805ec40 tx=0x7f88c8017000).stop 2023-10-18T16:08:04.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.762+0000 7f88dde67700 1 -- 172.21.15.88:0/3816563702 shutdown_connections 2023-10-18T16:08:04.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.762+0000 7f88dde67700 1 --2- 172.21.15.88:0/3816563702 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f88c405f150 0x7f88c4061610 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:04.768 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.762+0000 7f88dde67700 1 --2- 172.21.15.88:0/3816563702 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f88d8153b50 0x7f88d80c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:04.768 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.762+0000 7f88dde67700 1 --2- 172.21.15.88:0/3816563702 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f88d8063d70 0x7f88d80be520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:04.768 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.762+0000 7f88dde67700 1 --2- 172.21.15.88:0/3816563702 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f88d80c6a60 0x7f88d80c4f20 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:04.768 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.762+0000 7f88dde67700 1 -- 172.21.15.88:0/3816563702 >> 172.21.15.88:0/3816563702 conn(0x7f88d805e050 msgr2=0x7f88d80c0fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:08:04.768 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.762+0000 7f88dde67700 1 -- 172.21.15.88:0/3816563702 shutdown_connections 2023-10-18T16:08:04.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:04.762+0000 7f88dde67700 1 -- 172.21.15.88:0/3816563702 wait complete. 2023-10-18T16:08:04.777 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:08:07.778 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:08:07.905 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:07.902+0000 7efd3dfab700 1 Processor -- start 2023-10-18T16:08:07.906 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:07.902+0000 7efd3dfab700 1 -- start start 2023-10-18T16:08:07.908 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:07.902+0000 7efd3dfab700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7efd3814d8c0 0x7efd38064440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:07.908 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:07.902+0000 7efd3dfab700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd38064150 con 0x7efd38065360 2023-10-18T16:08:07.908 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:07.902+0000 7efd3dfab700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd38153a40 con 0x7efd380a9050 2023-10-18T16:08:07.908 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:07.902+0000 7efd3dfab700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd38063e70 con 0x7efd3814d8c0 2023-10-18T16:08:07.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:07.903+0000 7efd377fe700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7efd3814d8c0 0x7efd38064440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:07.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:07.903+0000 7efd37fff700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7efd38065360 0x7efd3815acb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:49588/0 (socket says 172.21.15.88:49588) 2023-10-18T16:08:07.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:07.903+0000 7efd377fe700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7efd3814d8c0 0x7efd38064440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40240/0 (socket says 172.21.15.88:40240) 2023-10-18T16:08:07.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:07.903+0000 7efd37fff700 1 -- 172.21.15.88:0/4199119384 learned_addr learned my addr 172.21.15.88:0/4199119384 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:08:07.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:07.903+0000 7efd377fe700 1 -- 172.21.15.88:0/4199119384 >> v1:172.21.15.151:6789/0 conn(0x7efd380a9050 legacy=0x7efd38153ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:08:07.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:07.903+0000 7efd377fe700 1 -- 172.21.15.88:0/4199119384 >> v1:172.21.15.88:6789/0 conn(0x7efd38065360 legacy=0x7efd3815acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:07.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:07.903+0000 7efd377fe700 1 -- 172.21.15.88:0/4199119384 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd380c72c0 con 0x7efd3814d8c0 2023-10-18T16:08:07.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:07.903+0000 7efd367fc700 1 -- 172.21.15.88:0/4199119384 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1398748646 0 0) 0x7efd38064150 con 0x7efd38065360 2023-10-18T16:08:07.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:07.903+0000 7efd377fe700 1 --2- 172.21.15.88:0/4199119384 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd3814d8c0 0x7efd38064440 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7efd28009e60 tx=0x7efd28009780).ready entity=mon.0 client_cookie=b7101ca12951b427 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:08:07.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:07.903+0000 7efd367fc700 1 -- 172.21.15.88:0/4199119384 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd2800e070 con 0x7efd3814d8c0 2023-10-18T16:08:07.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:07.903+0000 7efd367fc700 1 -- 172.21.15.88:0/4199119384 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd2800bbf0 con 0x7efd3814d8c0 2023-10-18T16:08:07.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:07.903+0000 7efd367fc700 1 -- 172.21.15.88:0/4199119384 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd28012410 con 0x7efd3814d8c0 2023-10-18T16:08:07.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:07.904+0000 7efd3dfab700 1 -- 172.21.15.88:0/4199119384 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd3814d8c0 msgr2=0x7efd38064440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:07.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:07.904+0000 7efd3dfab700 1 --2- 172.21.15.88:0/4199119384 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd3814d8c0 0x7efd38064440 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7efd28009e60 tx=0x7efd28009780).stop 2023-10-18T16:08:07.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:07.904+0000 7efd3dfab700 1 -- 172.21.15.88:0/4199119384 shutdown_connections 2023-10-18T16:08:07.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:07.904+0000 7efd3dfab700 1 --2- 172.21.15.88:0/4199119384 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd3814d8c0 0x7efd38064440 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:07.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:07.904+0000 7efd3dfab700 1 -- 172.21.15.88:0/4199119384 >> 172.21.15.88:0/4199119384 conn(0x7efd3806a810 msgr2=0x7efd380c6a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:08:07.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:07.904+0000 7efd3dfab700 1 -- 172.21.15.88:0/4199119384 shutdown_connections 2023-10-18T16:08:07.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:07.904+0000 7efd3dfab700 1 -- 172.21.15.88:0/4199119384 wait complete. 2023-10-18T16:08:07.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:07.905+0000 7efd3dfab700 1 Processor -- start 2023-10-18T16:08:07.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:07.905+0000 7efd3dfab700 1 -- start start 2023-10-18T16:08:07.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:07.906+0000 7efd3dfab700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd38065360 0x7efd380bb860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:07.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:07.906+0000 7efd3dfab700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efd380a9050 0x7efd38148d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:07.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:07.906+0000 7efd3dfab700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efd3814d8c0 0x7efd38149220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:07.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:07.906+0000 7efd3dfab700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd38149710 con 0x7efd38065360 2023-10-18T16:08:07.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:07.906+0000 7efd3dfab700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd380c0a50 con 0x7efd3814d8c0 2023-10-18T16:08:07.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:07.906+0000 7efd3dfab700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd380befc0 con 0x7efd380a9050 2023-10-18T16:08:07.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:07.906+0000 7efd377fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd38065360 0x7efd380bb860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:07.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:07.906+0000 7efd377fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd38065360 0x7efd380bb860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40248/0 (socket says 172.21.15.88:40248) 2023-10-18T16:08:07.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:07.906+0000 7efd377fe700 1 -- 172.21.15.88:0/2276994449 learned_addr learned my addr 172.21.15.88:0/2276994449 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:08:07.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:07.906+0000 7efd37fff700 1 --2- 172.21.15.88:0/2276994449 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efd3814d8c0 0x7efd38149220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:07.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:07.906+0000 7efd36ffd700 1 --2- 172.21.15.88:0/2276994449 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efd380a9050 0x7efd38148d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:07.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:07.906+0000 7efd377fe700 1 -- 172.21.15.88:0/2276994449 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efd380a9050 msgr2=0x7efd38148d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:07.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:07.906+0000 7efd377fe700 1 --2- 172.21.15.88:0/2276994449 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efd380a9050 0x7efd38148d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:07.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:07.906+0000 7efd377fe700 1 -- 172.21.15.88:0/2276994449 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efd3814d8c0 msgr2=0x7efd38149220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:07.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:07.906+0000 7efd377fe700 1 --2- 172.21.15.88:0/2276994449 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efd3814d8c0 0x7efd38149220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:07.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:07.906+0000 7efd377fe700 1 -- 172.21.15.88:0/2276994449 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd28009a20 con 0x7efd38065360 2023-10-18T16:08:07.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:07.906+0000 7efd37fff700 1 --2- 172.21.15.88:0/2276994449 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efd3814d8c0 0x7efd38149220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:08:07.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:07.906+0000 7efd377fe700 1 --2- 172.21.15.88:0/2276994449 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd38065360 0x7efd380bb860 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7efd38071c10 tx=0x7efd28009780).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:08:07.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:07.906+0000 7efd1f7fe700 1 -- 172.21.15.88:0/2276994449 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd2801d630 con 0x7efd38065360 2023-10-18T16:08:07.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:07.906+0000 7efd1f7fe700 1 -- 172.21.15.88:0/2276994449 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd2801d790 con 0x7efd38065360 2023-10-18T16:08:07.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:07.906+0000 7efd1f7fe700 1 -- 172.21.15.88:0/2276994449 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd2801db50 con 0x7efd38065360 2023-10-18T16:08:07.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:07.906+0000 7efd3dfab700 1 -- 172.21.15.88:0/2276994449 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd380bdd30 con 0x7efd38065360 2023-10-18T16:08:07.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:07.907+0000 7efd3dfab700 1 -- 172.21.15.88:0/2276994449 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd380c0120 con 0x7efd38065360 2023-10-18T16:08:07.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:07.907+0000 7efd3dfab700 1 -- 172.21.15.88:0/2276994449 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd3806c4e0 con 0x7efd38065360 2023-10-18T16:08:07.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:07.908+0000 7efd1f7fe700 1 -- 172.21.15.88:0/2276994449 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7efd28026410 con 0x7efd38065360 2023-10-18T16:08:07.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:07.908+0000 7efd1f7fe700 1 --2- 172.21.15.88:0/2276994449 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7efd2405f180 0x7efd24061640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:07.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:07.908+0000 7efd1f7fe700 1 -- 172.21.15.88:0/2276994449 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6121+0+0 (secure 0 0 0) 0x7efd2807bfb0 con 0x7efd38065360 2023-10-18T16:08:07.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:07.908+0000 7efd36ffd700 1 --2- 172.21.15.88:0/2276994449 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7efd2405f180 0x7efd24061640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:07.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:07.909+0000 7efd36ffd700 1 --2- 172.21.15.88:0/2276994449 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7efd2405f180 0x7efd24061640 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7efd380c2eb0 tx=0x7efd2c009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:08:07.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:07.912+0000 7efd1f7fe700 1 -- 172.21.15.88:0/2276994449 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7efd280aaac0 con 0x7efd38065360 2023-10-18T16:08:08.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:08.116+0000 7efd3dfab700 1 -- 172.21.15.88:0/2276994449 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7efd380aa400 con 0x7efd38065360 2023-10-18T16:08:08.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:08.116+0000 7efd1f7fe700 1 -- 172.21.15.88:0/2276994449 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7efd2804d220 con 0x7efd38065360 2023-10-18T16:08:08.120 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:08:08.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:08.118+0000 7efd3dfab700 1 -- 172.21.15.88:0/2276994449 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7efd2405f180 msgr2=0x7efd24061640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:08.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:08.118+0000 7efd3dfab700 1 --2- 172.21.15.88:0/2276994449 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7efd2405f180 0x7efd24061640 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7efd380c2eb0 tx=0x7efd2c009040).stop 2023-10-18T16:08:08.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:08.118+0000 7efd3dfab700 1 -- 172.21.15.88:0/2276994449 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd38065360 msgr2=0x7efd380bb860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:08.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:08.118+0000 7efd3dfab700 1 --2- 172.21.15.88:0/2276994449 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd38065360 0x7efd380bb860 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7efd38071c10 tx=0x7efd28009780).stop 2023-10-18T16:08:08.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:08.118+0000 7efd3dfab700 1 -- 172.21.15.88:0/2276994449 shutdown_connections 2023-10-18T16:08:08.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:08.118+0000 7efd3dfab700 1 --2- 172.21.15.88:0/2276994449 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7efd2405f180 0x7efd24061640 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:08.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:08.118+0000 7efd3dfab700 1 --2- 172.21.15.88:0/2276994449 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efd3814d8c0 0x7efd38149220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:08.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:08.118+0000 7efd3dfab700 1 --2- 172.21.15.88:0/2276994449 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd38065360 0x7efd380bb860 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:08.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:08.118+0000 7efd3dfab700 1 --2- 172.21.15.88:0/2276994449 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efd380a9050 0x7efd38148d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:08.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:08.118+0000 7efd3dfab700 1 -- 172.21.15.88:0/2276994449 >> 172.21.15.88:0/2276994449 conn(0x7efd3806a810 msgr2=0x7efd380c14a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:08:08.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:08.118+0000 7efd3dfab700 1 -- 172.21.15.88:0/2276994449 shutdown_connections 2023-10-18T16:08:08.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:08.118+0000 7efd3dfab700 1 -- 172.21.15.88:0/2276994449 wait complete. 2023-10-18T16:08:08.133 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:08:08.236 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-18T16:08:08.239 DEBUG:teuthology.orchestra.run.smithi151:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-18T16:08:10.021 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~20s 2023-10-18T16:08:11.134 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:08:11.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.255+0000 7ff115024700 1 Processor -- start 2023-10-18T16:08:11.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.255+0000 7ff115024700 1 -- start start 2023-10-18T16:08:11.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.256+0000 7ff115024700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff110063d70 0x7ff1100c6e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:11.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.256+0000 7ff115024700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff110151020 con 0x7ff1100c6a60 2023-10-18T16:08:11.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.256+0000 7ff115024700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff110150b90 con 0x7ff1100c2400 2023-10-18T16:08:11.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.256+0000 7ff115024700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff110150090 con 0x7ff110063d70 2023-10-18T16:08:11.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.256+0000 7ff10ffff700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7ff1100c6a60 0x7ff110153660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:49590/0 (socket says 172.21.15.88:49590) 2023-10-18T16:08:11.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.256+0000 7ff10ffff700 1 -- 172.21.15.88:0/221094952 learned_addr learned my addr 172.21.15.88:0/221094952 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:08:11.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.256+0000 7ff10f7fe700 1 --2- 172.21.15.88:0/221094952 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff110063d70 0x7ff1100c6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:11.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.256+0000 7ff10effd700 1 -- 172.21.15.88:0/221094952 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1208266997 0 0) 0x7ff100001110 con 0x7ff1100c6a60 2023-10-18T16:08:11.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.256+0000 7ff10effd700 1 -- 172.21.15.88:0/221094952 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0f80033f0 con 0x7ff1100c6a60 2023-10-18T16:08:11.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.257+0000 7ff10effd700 1 -- 172.21.15.88:0/221094952 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2290014812 0 0) 0x7ff0f80033f0 con 0x7ff1100c6a60 2023-10-18T16:08:11.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.257+0000 7ff10effd700 1 -- 172.21.15.88:0/221094952 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff110063d70 msgr2=0x7ff1100c6e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:08:11.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.257+0000 7ff10effd700 1 --2- 172.21.15.88:0/221094952 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff110063d70 0x7ff1100c6e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:11.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.257+0000 7ff10effd700 1 -- 172.21.15.88:0/221094952 >> v1:172.21.15.151:6789/0 conn(0x7ff1100c2400 legacy=0x7ff1100c3300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:11.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.257+0000 7ff10effd700 1 -- 172.21.15.88:0/221094952 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1100c2f70 con 0x7ff1100c6a60 2023-10-18T16:08:11.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.257+0000 7ff10effd700 1 -- 172.21.15.88:0/221094952 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441715075 0 0) 0x7ff100001600 con 0x7ff1100c6a60 2023-10-18T16:08:11.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.257+0000 7ff10effd700 1 -- 172.21.15.88:0/221094952 >> v1:172.21.15.88:6789/0 conn(0x7ff1100c6a60 legacy=0x7ff110153660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:11.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.257+0000 7ff10effd700 1 --2- 172.21.15.88:0/221094952 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff110063d70 0x7ff0f8003910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:11.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.257+0000 7ff10effd700 1 --2- 172.21.15.88:0/221094952 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff0f8003e00 0x7ff0f8004690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:11.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.257+0000 7ff10effd700 1 --2- 172.21.15.88:0/221094952 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0f8004b80 0x7ff0f8005000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:11.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.257+0000 7ff10effd700 1 -- 172.21.15.88:0/221094952 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f800b8e0 con 0x7ff0f8004b80 2023-10-18T16:08:11.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.257+0000 7ff10effd700 1 -- 172.21.15.88:0/221094952 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f800ba20 con 0x7ff0f8003e00 2023-10-18T16:08:11.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.257+0000 7ff10effd700 1 -- 172.21.15.88:0/221094952 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f800bbe0 con 0x7ff110063d70 2023-10-18T16:08:11.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.257+0000 7ff114823700 1 --2- 172.21.15.88:0/221094952 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0f8004b80 0x7ff0f8005000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:11.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.257+0000 7ff10ffff700 1 --2- 172.21.15.88:0/221094952 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff110063d70 0x7ff0f8003910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:11.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.257+0000 7ff10f7fe700 1 --2- 172.21.15.88:0/221094952 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff0f8003e00 0x7ff0f8004690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:11.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.257+0000 7ff114823700 1 -- 172.21.15.88:0/221094952 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff110063d70 msgr2=0x7ff0f8003910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:08:11.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.257+0000 7ff114823700 1 --2- 172.21.15.88:0/221094952 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff110063d70 0x7ff0f8003910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:11.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.257+0000 7ff114823700 1 -- 172.21.15.88:0/221094952 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff0f8003e00 msgr2=0x7ff0f8004690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:08:11.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.257+0000 7ff114823700 1 --2- 172.21.15.88:0/221094952 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff0f8003e00 0x7ff0f8004690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:11.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.257+0000 7ff114823700 1 -- 172.21.15.88:0/221094952 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0f80033f0 con 0x7ff0f8004b80 2023-10-18T16:08:11.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.257+0000 7ff114823700 1 --2- 172.21.15.88:0/221094952 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0f8004b80 0x7ff0f8005000 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7ff108001f20 tx=0x7ff108007f10).ready entity=mon.0 client_cookie=7393b7daffc9f51a server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:08:11.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.257+0000 7ff10effd700 1 -- 172.21.15.88:0/221094952 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1080017e0 con 0x7ff0f8004b80 2023-10-18T16:08:11.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.257+0000 7ff10effd700 1 -- 172.21.15.88:0/221094952 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff10800c6d0 con 0x7ff0f8004b80 2023-10-18T16:08:11.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.257+0000 7ff10effd700 1 -- 172.21.15.88:0/221094952 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff10800ca50 con 0x7ff0f8004b80 2023-10-18T16:08:11.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.258+0000 7ff115024700 1 -- 172.21.15.88:0/221094952 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0f8004b80 msgr2=0x7ff0f8005000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:11.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.258+0000 7ff115024700 1 --2- 172.21.15.88:0/221094952 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0f8004b80 0x7ff0f8005000 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7ff108001f20 tx=0x7ff108007f10).stop 2023-10-18T16:08:11.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.258+0000 7ff114823700 1 -- 172.21.15.88:0/221094952 reap_dead start 2023-10-18T16:08:11.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.258+0000 7ff115024700 1 -- 172.21.15.88:0/221094952 shutdown_connections 2023-10-18T16:08:11.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.258+0000 7ff115024700 1 -- 172.21.15.88:0/221094952 >> 172.21.15.88:0/221094952 conn(0x7ff11005e050 msgr2=0x7ff11005e460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:08:11.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.258+0000 7ff115024700 1 -- 172.21.15.88:0/221094952 shutdown_connections 2023-10-18T16:08:11.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.258+0000 7ff115024700 1 -- 172.21.15.88:0/221094952 wait complete. 2023-10-18T16:08:11.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.259+0000 7ff115024700 1 Processor -- start 2023-10-18T16:08:11.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.259+0000 7ff115024700 1 -- start start 2023-10-18T16:08:11.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.260+0000 7ff115024700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff1100c1680 0x7ff1100c1a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:11.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.260+0000 7ff115024700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff110148dd0 0x7ff11015ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:11.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.260+0000 7ff115024700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff1101604c0 0x7ff110147c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:11.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.260+0000 7ff115024700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff110151490 con 0x7ff110148dd0 2023-10-18T16:08:11.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.260+0000 7ff115024700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1101640c0 con 0x7ff1100c1680 2023-10-18T16:08:11.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.260+0000 7ff115024700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff110148af0 con 0x7ff1101604c0 2023-10-18T16:08:11.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.260+0000 7ff10f7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff110148dd0 0x7ff11015ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:11.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.260+0000 7ff10f7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff110148dd0 0x7ff11015ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40266/0 (socket says 172.21.15.88:40266) 2023-10-18T16:08:11.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.260+0000 7ff10f7fe700 1 -- 172.21.15.88:0/231016794 learned_addr learned my addr 172.21.15.88:0/231016794 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:08:11.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.260+0000 7ff10ffff700 1 --2- 172.21.15.88:0/231016794 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff1100c1680 0x7ff1100c1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:11.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.260+0000 7ff114823700 1 --2- 172.21.15.88:0/231016794 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff1101604c0 0x7ff110147c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:11.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.260+0000 7ff10f7fe700 1 -- 172.21.15.88:0/231016794 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff1101604c0 msgr2=0x7ff110147c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:11.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.260+0000 7ff10f7fe700 1 --2- 172.21.15.88:0/231016794 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff1101604c0 0x7ff110147c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:11.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.260+0000 7ff10f7fe700 1 -- 172.21.15.88:0/231016794 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff1100c1680 msgr2=0x7ff1100c1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:11.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.260+0000 7ff10f7fe700 1 --2- 172.21.15.88:0/231016794 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff1100c1680 0x7ff1100c1a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:11.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.260+0000 7ff10f7fe700 1 -- 172.21.15.88:0/231016794 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1101516c0 con 0x7ff110148dd0 2023-10-18T16:08:11.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.261+0000 7ff10f7fe700 1 --2- 172.21.15.88:0/231016794 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff110148dd0 0x7ff11015ffd0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7ff1040119c0 tx=0x7ff104009060).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:08:11.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.261+0000 7ff0f7fff700 1 -- 172.21.15.88:0/231016794 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff10400f8e0 con 0x7ff110148dd0 2023-10-18T16:08:11.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.261+0000 7ff0f7fff700 1 -- 172.21.15.88:0/231016794 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff10400fa40 con 0x7ff110148dd0 2023-10-18T16:08:11.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.261+0000 7ff0f7fff700 1 -- 172.21.15.88:0/231016794 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff10400fdc0 con 0x7ff110148dd0 2023-10-18T16:08:11.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.261+0000 7ff115024700 1 -- 172.21.15.88:0/231016794 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff110163d50 con 0x7ff110148dd0 2023-10-18T16:08:11.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.261+0000 7ff115024700 1 -- 172.21.15.88:0/231016794 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1100c4b30 con 0x7ff110148dd0 2023-10-18T16:08:11.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.262+0000 7ff115024700 1 -- 172.21.15.88:0/231016794 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff11015fdf0 con 0x7ff110148dd0 2023-10-18T16:08:11.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.262+0000 7ff0f7fff700 1 -- 172.21.15.88:0/231016794 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7ff104014650 con 0x7ff110148dd0 2023-10-18T16:08:11.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.262+0000 7ff0f7fff700 1 --2- 172.21.15.88:0/231016794 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7ff0fc05f150 0x7ff0fc061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:11.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.263+0000 7ff0f7fff700 1 -- 172.21.15.88:0/231016794 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6121+0+0 (secure 0 0 0) 0x7ff10407add0 con 0x7ff110148dd0 2023-10-18T16:08:11.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.263+0000 7ff10ffff700 1 --2- 172.21.15.88:0/231016794 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7ff0fc05f150 0x7ff0fc061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:11.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.263+0000 7ff10ffff700 1 --2- 172.21.15.88:0/231016794 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7ff0fc05f150 0x7ff0fc061610 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7ff100001e10 tx=0x7ff100009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:08:11.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.267+0000 7ff0f7fff700 1 -- 172.21.15.88:0/231016794 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7ff10404c0a0 con 0x7ff110148dd0 2023-10-18T16:08:11.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.466+0000 7ff115024700 1 -- 172.21.15.88:0/231016794 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff1101508e0 con 0x7ff110148dd0 2023-10-18T16:08:11.471 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.466+0000 7ff0f7fff700 1 -- 172.21.15.88:0/231016794 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7ff10407c020 con 0x7ff110148dd0 2023-10-18T16:08:11.471 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:08:11.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.468+0000 7ff115024700 1 -- 172.21.15.88:0/231016794 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7ff0fc05f150 msgr2=0x7ff0fc061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:11.473 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.468+0000 7ff115024700 1 --2- 172.21.15.88:0/231016794 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7ff0fc05f150 0x7ff0fc061610 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7ff100001e10 tx=0x7ff100009040).stop 2023-10-18T16:08:11.473 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.468+0000 7ff115024700 1 -- 172.21.15.88:0/231016794 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff110148dd0 msgr2=0x7ff11015ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:11.473 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.468+0000 7ff115024700 1 --2- 172.21.15.88:0/231016794 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff110148dd0 0x7ff11015ffd0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7ff1040119c0 tx=0x7ff104009060).stop 2023-10-18T16:08:11.473 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.468+0000 7ff115024700 1 -- 172.21.15.88:0/231016794 shutdown_connections 2023-10-18T16:08:11.473 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.468+0000 7ff115024700 1 --2- 172.21.15.88:0/231016794 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7ff0fc05f150 0x7ff0fc061610 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:11.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.468+0000 7ff115024700 1 --2- 172.21.15.88:0/231016794 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff1101604c0 0x7ff110147c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:11.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.468+0000 7ff115024700 1 --2- 172.21.15.88:0/231016794 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff1100c1680 0x7ff1100c1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:11.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.468+0000 7ff115024700 1 --2- 172.21.15.88:0/231016794 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff110148dd0 0x7ff11015ffd0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:11.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.468+0000 7ff115024700 1 -- 172.21.15.88:0/231016794 >> 172.21.15.88:0/231016794 conn(0x7ff11005e050 msgr2=0x7ff11014c130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:08:11.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.468+0000 7ff115024700 1 -- 172.21.15.88:0/231016794 shutdown_connections 2023-10-18T16:08:11.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:11.468+0000 7ff115024700 1 -- 172.21.15.88:0/231016794 wait complete. 2023-10-18T16:08:11.484 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:08:14.485 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:08:14.609 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.605+0000 7f734dd68700 1 Processor -- start 2023-10-18T16:08:14.610 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.605+0000 7f734dd68700 1 -- start start 2023-10-18T16:08:14.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.606+0000 7f734dd68700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7348153010 0x7f73481533f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:14.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.606+0000 7f734dd68700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7348064d70 0x7f7348151ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:14.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.606+0000 7f734dd68700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73480a9670 con 0x7f73481523d0 2023-10-18T16:08:14.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.606+0000 7f734dd68700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f734814df50 con 0x7f7348064d70 2023-10-18T16:08:14.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.606+0000 7f734dd68700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7348151a50 con 0x7f7348153010 2023-10-18T16:08:14.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.606+0000 7f7346ffd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7348064d70 0x7f7348151ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:14.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.606+0000 7f7346ffd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7348064d70 0x7f7348151ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40270/0 (socket says 172.21.15.88:40270) 2023-10-18T16:08:14.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.606+0000 7f7346ffd700 1 -- 172.21.15.88:0/1033203328 learned_addr learned my addr 172.21.15.88:0/1033203328 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:08:14.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.607+0000 7f73477fe700 1 --2- 172.21.15.88:0/1033203328 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7348153010 0x7f73481533f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:14.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.607+0000 7f7346ffd700 1 -- 172.21.15.88:0/1033203328 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7348153010 msgr2=0x7f73481533f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:08:14.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.607+0000 7f7346ffd700 1 --2- 172.21.15.88:0/1033203328 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7348153010 0x7f73481533f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:14.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.607+0000 7f7346ffd700 1 -- 172.21.15.88:0/1033203328 >> v1:172.21.15.151:6789/0 conn(0x7f73481523d0 legacy=0x7f73480c36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:08:14.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.607+0000 7f7346ffd700 1 -- 172.21.15.88:0/1033203328 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73480a8af0 con 0x7f7348064d70 2023-10-18T16:08:14.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.607+0000 7f7346ffd700 1 --2- 172.21.15.88:0/1033203328 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7348064d70 0x7f7348151ee0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f733c006fd0 tx=0x7f733c0090f0).ready entity=mon.0 client_cookie=4c3e828906df2b68 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:08:14.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.607+0000 7f73467fc700 1 -- 172.21.15.88:0/1033203328 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f733c00b600 con 0x7f7348064d70 2023-10-18T16:08:14.620 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.607+0000 7f73467fc700 1 -- 172.21.15.88:0/1033203328 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f733c00b760 con 0x7f7348064d70 2023-10-18T16:08:14.620 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.607+0000 7f73467fc700 1 -- 172.21.15.88:0/1033203328 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f733c00bb00 con 0x7f7348064d70 2023-10-18T16:08:14.620 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.607+0000 7f734dd68700 1 -- 172.21.15.88:0/1033203328 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7348064d70 msgr2=0x7f7348151ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:14.620 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.608+0000 7f734dd68700 1 --2- 172.21.15.88:0/1033203328 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7348064d70 0x7f7348151ee0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f733c006fd0 tx=0x7f733c0090f0).stop 2023-10-18T16:08:14.620 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.608+0000 7f734dd68700 1 -- 172.21.15.88:0/1033203328 shutdown_connections 2023-10-18T16:08:14.620 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.608+0000 7f734dd68700 1 --2- 172.21.15.88:0/1033203328 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7348153010 0x7f73481533f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:14.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.608+0000 7f734dd68700 1 --2- 172.21.15.88:0/1033203328 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7348064d70 0x7f7348151ee0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:14.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.608+0000 7f734dd68700 1 -- 172.21.15.88:0/1033203328 >> 172.21.15.88:0/1033203328 conn(0x7f7348069d80 msgr2=0x7f734806a190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:08:14.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.608+0000 7f734dd68700 1 -- 172.21.15.88:0/1033203328 shutdown_connections 2023-10-18T16:08:14.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.608+0000 7f734dd68700 1 -- 172.21.15.88:0/1033203328 wait complete. 2023-10-18T16:08:14.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.609+0000 7f734dd68700 1 Processor -- start 2023-10-18T16:08:14.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.609+0000 7f734dd68700 1 -- start start 2023-10-18T16:08:14.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.610+0000 7f734dd68700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7348064d70 0x7f73480a5840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:14.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.610+0000 7f734dd68700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73481523d0 0x7f734814e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:14.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.610+0000 7f734dd68700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7348153010 0x7f734814e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:14.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.610+0000 7f734dd68700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7348151300 con 0x7f73481523d0 2023-10-18T16:08:14.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.610+0000 7f734dd68700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73481506f0 con 0x7f7348064d70 2023-10-18T16:08:14.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.610+0000 7f734dd68700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73481509d0 con 0x7f7348153010 2023-10-18T16:08:14.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.610+0000 7f7346ffd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73481523d0 0x7f734814e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:14.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.610+0000 7f7346ffd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73481523d0 0x7f734814e2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40284/0 (socket says 172.21.15.88:40284) 2023-10-18T16:08:14.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.610+0000 7f7346ffd700 1 -- 172.21.15.88:0/2078648403 learned_addr learned my addr 172.21.15.88:0/2078648403 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:08:14.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.610+0000 7f73477fe700 1 --2- 172.21.15.88:0/2078648403 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7348064d70 0x7f73480a5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:14.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.610+0000 7f7347fff700 1 --2- 172.21.15.88:0/2078648403 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7348153010 0x7f734814e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:14.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.610+0000 7f7346ffd700 1 -- 172.21.15.88:0/2078648403 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7348153010 msgr2=0x7f734814e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:14.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.610+0000 7f7346ffd700 1 --2- 172.21.15.88:0/2078648403 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7348153010 0x7f734814e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:14.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.610+0000 7f7346ffd700 1 -- 172.21.15.88:0/2078648403 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7348064d70 msgr2=0x7f73480a5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:14.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.610+0000 7f7346ffd700 1 --2- 172.21.15.88:0/2078648403 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7348064d70 0x7f73480a5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:14.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.610+0000 7f7346ffd700 1 -- 172.21.15.88:0/2078648403 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f733c009370 con 0x7f73481523d0 2023-10-18T16:08:14.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.610+0000 7f7347fff700 1 --2- 172.21.15.88:0/2078648403 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7348153010 0x7f734814e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:08:14.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.610+0000 7f73477fe700 1 --2- 172.21.15.88:0/2078648403 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7348064d70 0x7f73480a5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:08:14.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.610+0000 7f7346ffd700 1 --2- 172.21.15.88:0/2078648403 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73481523d0 0x7f734814e2e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f7348084410 tx=0x7f733c017000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:08:14.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.610+0000 7f732f7fe700 1 -- 172.21.15.88:0/2078648403 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f733c00b8d0 con 0x7f73481523d0 2023-10-18T16:08:14.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.610+0000 7f732f7fe700 1 -- 172.21.15.88:0/2078648403 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f733c00ba30 con 0x7f73481523d0 2023-10-18T16:08:14.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.610+0000 7f734dd68700 1 -- 172.21.15.88:0/2078648403 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7348150cb0 con 0x7f73481523d0 2023-10-18T16:08:14.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.610+0000 7f732f7fe700 1 -- 172.21.15.88:0/2078648403 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f733c01c670 con 0x7f73481523d0 2023-10-18T16:08:14.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.610+0000 7f734dd68700 1 -- 172.21.15.88:0/2078648403 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7348163280 con 0x7f73481523d0 2023-10-18T16:08:14.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.611+0000 7f734dd68700 1 -- 172.21.15.88:0/2078648403 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7348151150 con 0x7f73481523d0 2023-10-18T16:08:14.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.612+0000 7f732f7fe700 1 -- 172.21.15.88:0/2078648403 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f733c01ca60 con 0x7f73481523d0 2023-10-18T16:08:14.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.612+0000 7f732f7fe700 1 --2- 172.21.15.88:0/2078648403 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f733405f150 0x7f7334061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:14.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.612+0000 7f732f7fe700 1 -- 172.21.15.88:0/2078648403 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6121+0+0 (secure 0 0 0) 0x7f733c07bfa0 con 0x7f73481523d0 2023-10-18T16:08:14.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.612+0000 7f73477fe700 1 --2- 172.21.15.88:0/2078648403 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f733405f150 0x7f7334061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:14.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.613+0000 7f73477fe700 1 --2- 172.21.15.88:0/2078648403 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f733405f150 0x7f7334061610 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f733800e420 tx=0x7f7338007650).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:08:14.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.616+0000 7f732f7fe700 1 -- 172.21.15.88:0/2078648403 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f733c0aaac0 con 0x7f73481523d0 2023-10-18T16:08:14.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.817+0000 7f734dd68700 1 -- 172.21.15.88:0/2078648403 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f734806c4e0 con 0x7f73481523d0 2023-10-18T16:08:14.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.817+0000 7f732f7fe700 1 -- 172.21.15.88:0/2078648403 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7f733c02a070 con 0x7f73481523d0 2023-10-18T16:08:14.821 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:08:14.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.819+0000 7f734dd68700 1 -- 172.21.15.88:0/2078648403 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f733405f150 msgr2=0x7f7334061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:14.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.819+0000 7f734dd68700 1 --2- 172.21.15.88:0/2078648403 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f733405f150 0x7f7334061610 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f733800e420 tx=0x7f7338007650).stop 2023-10-18T16:08:14.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.819+0000 7f734dd68700 1 -- 172.21.15.88:0/2078648403 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73481523d0 msgr2=0x7f734814e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:14.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.819+0000 7f734dd68700 1 --2- 172.21.15.88:0/2078648403 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73481523d0 0x7f734814e2e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f7348084410 tx=0x7f733c017000).stop 2023-10-18T16:08:14.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.819+0000 7f734dd68700 1 -- 172.21.15.88:0/2078648403 shutdown_connections 2023-10-18T16:08:14.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.819+0000 7f734dd68700 1 --2- 172.21.15.88:0/2078648403 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f733405f150 0x7f7334061610 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:14.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.819+0000 7f734dd68700 1 --2- 172.21.15.88:0/2078648403 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7348153010 0x7f734814e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:14.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.819+0000 7f734dd68700 1 --2- 172.21.15.88:0/2078648403 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7348064d70 0x7f73480a5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:14.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.819+0000 7f734dd68700 1 --2- 172.21.15.88:0/2078648403 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73481523d0 0x7f734814e2e0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:14.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.819+0000 7f734dd68700 1 -- 172.21.15.88:0/2078648403 >> 172.21.15.88:0/2078648403 conn(0x7f7348069d80 msgr2=0x7f73480c5b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:08:14.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.819+0000 7f734dd68700 1 -- 172.21.15.88:0/2078648403 shutdown_connections 2023-10-18T16:08:14.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:14.819+0000 7f734dd68700 1 -- 172.21.15.88:0/2078648403 wait complete. 2023-10-18T16:08:14.834 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:08:16.834 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~27s 2023-10-18T16:08:17.835 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:08:17.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:17.956+0000 7feb3944e700 1 Processor -- start 2023-10-18T16:08:17.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:17.956+0000 7feb3944e700 1 -- start start 2023-10-18T16:08:17.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:17.957+0000 7feb3944e700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7feb3414d8c0 0x7feb34064440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:17.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:17.957+0000 7feb3944e700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7feb340a9050 0x7feb34064930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:17.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:17.957+0000 7feb3944e700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb34064150 con 0x7feb34065360 2023-10-18T16:08:17.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:17.957+0000 7feb3944e700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb34153a40 con 0x7feb340a9050 2023-10-18T16:08:17.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:17.957+0000 7feb3944e700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb34063e70 con 0x7feb3414d8c0 2023-10-18T16:08:17.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:17.957+0000 7feb2bfff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7feb340a9050 0x7feb34064930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:17.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:17.957+0000 7feb2bfff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7feb340a9050 0x7feb34064930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56062/0 (socket says 172.21.15.88:56062) 2023-10-18T16:08:17.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:17.957+0000 7feb2bfff700 1 -- 172.21.15.88:0/3942275177 learned_addr learned my addr 172.21.15.88:0/3942275177 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:08:17.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:17.957+0000 7feb32ffd700 1 --2- 172.21.15.88:0/3942275177 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7feb3414d8c0 0x7feb34064440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:17.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:17.957+0000 7feb2bfff700 1 -- 172.21.15.88:0/3942275177 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7feb3414d8c0 msgr2=0x7feb34064440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:08:17.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:17.957+0000 7feb2bfff700 1 --2- 172.21.15.88:0/3942275177 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7feb3414d8c0 0x7feb34064440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:17.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:17.957+0000 7feb2bfff700 1 -- 172.21.15.88:0/3942275177 >> v1:172.21.15.151:6789/0 conn(0x7feb34065360 legacy=0x7feb34153ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:08:17.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:17.957+0000 7feb2bfff700 1 -- 172.21.15.88:0/3942275177 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb340c72c0 con 0x7feb340a9050 2023-10-18T16:08:17.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:17.957+0000 7feb32ffd700 1 --2- 172.21.15.88:0/3942275177 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7feb3414d8c0 0x7feb34064440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-10-18T16:08:17.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:17.957+0000 7feb2bfff700 1 --2- 172.21.15.88:0/3942275177 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feb340a9050 0x7feb34064930 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7feb1c006fd0 tx=0x7feb1c0090f0).ready entity=mon.0 client_cookie=5408d36ccebf414e server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:08:17.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:17.957+0000 7feb327fc700 1 -- 172.21.15.88:0/3942275177 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb1c00b600 con 0x7feb340a9050 2023-10-18T16:08:17.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:17.957+0000 7feb327fc700 1 -- 172.21.15.88:0/3942275177 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb1c00b760 con 0x7feb340a9050 2023-10-18T16:08:17.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:17.957+0000 7feb327fc700 1 -- 172.21.15.88:0/3942275177 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb1c00bb00 con 0x7feb340a9050 2023-10-18T16:08:17.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:17.958+0000 7feb3944e700 1 -- 172.21.15.88:0/3942275177 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feb340a9050 msgr2=0x7feb34064930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:17.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:17.958+0000 7feb3944e700 1 --2- 172.21.15.88:0/3942275177 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feb340a9050 0x7feb34064930 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7feb1c006fd0 tx=0x7feb1c0090f0).stop 2023-10-18T16:08:17.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:17.958+0000 7feb3944e700 1 -- 172.21.15.88:0/3942275177 shutdown_connections 2023-10-18T16:08:17.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:17.958+0000 7feb3944e700 1 --2- 172.21.15.88:0/3942275177 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7feb3414d8c0 0x7feb34064440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:17.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:17.958+0000 7feb3944e700 1 --2- 172.21.15.88:0/3942275177 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feb340a9050 0x7feb34064930 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:17.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:17.958+0000 7feb3944e700 1 -- 172.21.15.88:0/3942275177 >> 172.21.15.88:0/3942275177 conn(0x7feb3406a810 msgr2=0x7feb340c6a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:08:17.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:17.958+0000 7feb3944e700 1 -- 172.21.15.88:0/3942275177 shutdown_connections 2023-10-18T16:08:17.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:17.958+0000 7feb3944e700 1 -- 172.21.15.88:0/3942275177 wait complete. 2023-10-18T16:08:17.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:17.959+0000 7feb3944e700 1 Processor -- start 2023-10-18T16:08:17.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:17.959+0000 7feb3944e700 1 -- start start 2023-10-18T16:08:17.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:17.960+0000 7feb3944e700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7feb34065360 0x7feb341481e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:17.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:17.960+0000 7feb3944e700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feb340a9050 0x7feb340bce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:17.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:17.960+0000 7feb3944e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feb3414d8c0 0x7feb340bfb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:17.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:17.960+0000 7feb3944e700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb340baee0 con 0x7feb340a9050 2023-10-18T16:08:17.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:17.960+0000 7feb3944e700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb34160f10 con 0x7feb3414d8c0 2023-10-18T16:08:17.972 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:17.960+0000 7feb3944e700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb34160a20 con 0x7feb34065360 2023-10-18T16:08:17.972 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:17.960+0000 7feb2bfff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feb340a9050 0x7feb340bce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:17.972 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:17.960+0000 7feb2bfff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feb340a9050 0x7feb340bce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56076/0 (socket says 172.21.15.88:56076) 2023-10-18T16:08:17.972 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:17.960+0000 7feb2bfff700 1 -- 172.21.15.88:0/448569588 learned_addr learned my addr 172.21.15.88:0/448569588 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:08:17.972 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:17.960+0000 7feb32ffd700 1 --2- 172.21.15.88:0/448569588 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7feb34065360 0x7feb341481e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:17.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:17.960+0000 7feb337fe700 1 --2- 172.21.15.88:0/448569588 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feb3414d8c0 0x7feb340bfb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:17.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:17.960+0000 7feb2bfff700 1 -- 172.21.15.88:0/448569588 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7feb34065360 msgr2=0x7feb341481e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:17.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:17.960+0000 7feb2bfff700 1 --2- 172.21.15.88:0/448569588 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7feb34065360 0x7feb341481e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:17.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:17.960+0000 7feb2bfff700 1 -- 172.21.15.88:0/448569588 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feb3414d8c0 msgr2=0x7feb340bfb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:17.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:17.960+0000 7feb2bfff700 1 --2- 172.21.15.88:0/448569588 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feb3414d8c0 0x7feb340bfb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:17.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:17.960+0000 7feb2bfff700 1 -- 172.21.15.88:0/448569588 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb1c009370 con 0x7feb340a9050 2023-10-18T16:08:17.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:17.960+0000 7feb2bfff700 1 --2- 172.21.15.88:0/448569588 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feb340a9050 0x7feb340bce90 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7feb34071c10 tx=0x7feb1c017000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:08:17.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:17.961+0000 7feb2affd700 1 -- 172.21.15.88:0/448569588 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb1c00bec0 con 0x7feb340a9050 2023-10-18T16:08:17.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:17.961+0000 7feb2affd700 1 -- 172.21.15.88:0/448569588 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb1c00b8d0 con 0x7feb340a9050 2023-10-18T16:08:17.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:17.961+0000 7feb3944e700 1 -- 172.21.15.88:0/448569588 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb34164020 con 0x7feb340a9050 2023-10-18T16:08:17.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:17.961+0000 7feb2affd700 1 -- 172.21.15.88:0/448569588 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb1c024470 con 0x7feb340a9050 2023-10-18T16:08:17.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:17.961+0000 7feb3944e700 1 -- 172.21.15.88:0/448569588 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb341636f0 con 0x7feb340a9050 2023-10-18T16:08:17.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:17.962+0000 7feb3944e700 1 -- 172.21.15.88:0/448569588 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb341641b0 con 0x7feb340a9050 2023-10-18T16:08:17.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:17.962+0000 7feb2affd700 1 -- 172.21.15.88:0/448569588 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7feb1c021070 con 0x7feb340a9050 2023-10-18T16:08:17.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:17.962+0000 7feb2affd700 1 --2- 172.21.15.88:0/448569588 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7feb1405f150 0x7feb14061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:17.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:17.962+0000 7feb2affd700 1 -- 172.21.15.88:0/448569588 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6121+0+0 (secure 0 0 0) 0x7feb1c07bd80 con 0x7feb340a9050 2023-10-18T16:08:17.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:17.963+0000 7feb32ffd700 1 --2- 172.21.15.88:0/448569588 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7feb1405f150 0x7feb14061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:17.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:17.963+0000 7feb32ffd700 1 --2- 172.21.15.88:0/448569588 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7feb1405f150 0x7feb14061610 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7feb2400ab40 tx=0x7feb24008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:08:17.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:17.966+0000 7feb2affd700 1 -- 172.21.15.88:0/448569588 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7feb1c04d050 con 0x7feb340a9050 2023-10-18T16:08:18.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:18.167+0000 7feb3944e700 1 -- 172.21.15.88:0/448569588 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7feb3406c4e0 con 0x7feb340a9050 2023-10-18T16:08:18.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:18.168+0000 7feb2affd700 1 -- 172.21.15.88:0/448569588 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7feb1c050670 con 0x7feb340a9050 2023-10-18T16:08:18.172 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:08:18.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:18.169+0000 7feb3944e700 1 -- 172.21.15.88:0/448569588 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7feb1405f150 msgr2=0x7feb14061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:18.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:18.169+0000 7feb3944e700 1 --2- 172.21.15.88:0/448569588 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7feb1405f150 0x7feb14061610 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7feb2400ab40 tx=0x7feb24008040).stop 2023-10-18T16:08:18.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:18.169+0000 7feb3944e700 1 -- 172.21.15.88:0/448569588 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feb340a9050 msgr2=0x7feb340bce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:18.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:18.169+0000 7feb3944e700 1 --2- 172.21.15.88:0/448569588 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feb340a9050 0x7feb340bce90 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7feb34071c10 tx=0x7feb1c017000).stop 2023-10-18T16:08:18.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:18.169+0000 7feb3944e700 1 -- 172.21.15.88:0/448569588 shutdown_connections 2023-10-18T16:08:18.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:18.169+0000 7feb3944e700 1 --2- 172.21.15.88:0/448569588 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7feb1405f150 0x7feb14061610 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:18.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:18.169+0000 7feb3944e700 1 --2- 172.21.15.88:0/448569588 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feb3414d8c0 0x7feb340bfb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:18.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:18.169+0000 7feb3944e700 1 --2- 172.21.15.88:0/448569588 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7feb34065360 0x7feb341481e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:18.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:18.169+0000 7feb3944e700 1 --2- 172.21.15.88:0/448569588 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feb340a9050 0x7feb340bce90 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:18.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:18.169+0000 7feb3944e700 1 -- 172.21.15.88:0/448569588 >> 172.21.15.88:0/448569588 conn(0x7feb3406a810 msgr2=0x7feb340c14a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:08:18.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:18.169+0000 7feb3944e700 1 -- 172.21.15.88:0/448569588 shutdown_connections 2023-10-18T16:08:18.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:18.169+0000 7feb3944e700 1 -- 172.21.15.88:0/448569588 wait complete. 2023-10-18T16:08:18.185 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:08:21.185 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:08:21.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.304+0000 7fddb759e700 1 Processor -- start 2023-10-18T16:08:21.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.304+0000 7fddb759e700 1 -- start start 2023-10-18T16:08:21.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.305+0000 7fddb759e700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fddb80c6a60 0x7fddb80c6e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:21.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.305+0000 7fddb759e700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fddb8063d70 0x7fddb8153660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:21.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.305+0000 7fddb759e700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fddb8153b50 0x7fddb80c2400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:21.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.305+0000 7fddb759e700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddb8150090 con 0x7fddb8063d70 2023-10-18T16:08:21.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.305+0000 7fddb759e700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddb8151020 con 0x7fddb80c6a60 2023-10-18T16:08:21.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.305+0000 7fddb759e700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddb8150b90 con 0x7fddb8153b50 2023-10-18T16:08:21.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.305+0000 7fddb5d9b700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fddb8063d70 0x7fddb8153660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:21.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.305+0000 7fddb5d9b700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fddb8063d70 0x7fddb8153660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56088/0 (socket says 172.21.15.88:56088) 2023-10-18T16:08:21.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.305+0000 7fddb5d9b700 1 -- 172.21.15.88:0/2188237074 learned_addr learned my addr 172.21.15.88:0/2188237074 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:08:21.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.305+0000 7fddb659c700 1 --2- 172.21.15.88:0/2188237074 >> v2:172.21.15.151:3300/0 conn(0x7fddb80c6a60 0x7fddb80c6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:21.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.305+0000 7fddb6d9d700 1 --2- 172.21.15.88:0/2188237074 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fddb8153b50 0x7fddb80c2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:21.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.305+0000 7fddb5d9b700 1 -- 172.21.15.88:0/2188237074 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fddb8153b50 msgr2=0x7fddb80c2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:08:21.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.305+0000 7fddb5d9b700 1 --2- 172.21.15.88:0/2188237074 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fddb8153b50 0x7fddb80c2400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:21.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.305+0000 7fddb5d9b700 1 -- 172.21.15.88:0/2188237074 >> v2:172.21.15.151:3300/0 conn(0x7fddb80c6a60 msgr2=0x7fddb80c6e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:08:21.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.305+0000 7fddb5d9b700 1 --2- 172.21.15.88:0/2188237074 >> v2:172.21.15.151:3300/0 conn(0x7fddb80c6a60 0x7fddb80c6e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:21.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.305+0000 7fddb5d9b700 1 -- 172.21.15.88:0/2188237074 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddb80c2f70 con 0x7fddb8063d70 2023-10-18T16:08:21.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.305+0000 7fddb6d9d700 1 --2- 172.21.15.88:0/2188237074 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fddb8153b50 0x7fddb80c2400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:08:21.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.306+0000 7fddb5d9b700 1 --2- 172.21.15.88:0/2188237074 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fddb8063d70 0x7fddb8153660 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fddb80c43c0 tx=0x7fddac0090f0).ready entity=mon.0 client_cookie=34e7782abd6776ff server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:08:21.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.306+0000 7fddb559a700 1 -- 172.21.15.88:0/2188237074 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddac00b520 con 0x7fddb8063d70 2023-10-18T16:08:21.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.306+0000 7fddb559a700 1 -- 172.21.15.88:0/2188237074 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddac00b680 con 0x7fddb8063d70 2023-10-18T16:08:21.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.306+0000 7fddb559a700 1 -- 172.21.15.88:0/2188237074 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddac00ba00 con 0x7fddb8063d70 2023-10-18T16:08:21.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.306+0000 7fddb759e700 1 -- 172.21.15.88:0/2188237074 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fddb8063d70 msgr2=0x7fddb8153660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:21.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.306+0000 7fddb759e700 1 --2- 172.21.15.88:0/2188237074 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fddb8063d70 0x7fddb8153660 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fddb80c43c0 tx=0x7fddac0090f0).stop 2023-10-18T16:08:21.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.306+0000 7fddb759e700 1 -- 172.21.15.88:0/2188237074 shutdown_connections 2023-10-18T16:08:21.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.306+0000 7fddb759e700 1 --2- 172.21.15.88:0/2188237074 >> v2:172.21.15.151:3300/0 conn(0x7fddb80c6a60 0x7fddb80c6e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:21.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.306+0000 7fddb759e700 1 --2- 172.21.15.88:0/2188237074 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fddb8153b50 0x7fddb80c2400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:21.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.306+0000 7fddb759e700 1 --2- 172.21.15.88:0/2188237074 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fddb8063d70 0x7fddb8153660 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:21.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.306+0000 7fddb759e700 1 -- 172.21.15.88:0/2188237074 >> 172.21.15.88:0/2188237074 conn(0x7fddb805e050 msgr2=0x7fddb805e460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:08:21.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.306+0000 7fddb759e700 1 -- 172.21.15.88:0/2188237074 shutdown_connections 2023-10-18T16:08:21.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.306+0000 7fddb759e700 1 -- 172.21.15.88:0/2188237074 wait complete. 2023-10-18T16:08:21.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.307+0000 7fddb759e700 1 Processor -- start 2023-10-18T16:08:21.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.308+0000 7fddb759e700 1 -- start start 2023-10-18T16:08:21.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.308+0000 7fddb759e700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fddb8063d70 0x7fddb8160510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:21.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.308+0000 7fddb759e700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fddb80c6a60 0x7fddb814f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:21.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.308+0000 7fddb759e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fddb8153b50 0x7fddb8152f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:21.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.308+0000 7fddb759e700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddb81542a0 con 0x7fddb8063d70 2023-10-18T16:08:21.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.308+0000 7fddb759e700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddb8152530 con 0x7fddb8153b50 2023-10-18T16:08:21.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.308+0000 7fddb759e700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddb8152af0 con 0x7fddb80c6a60 2023-10-18T16:08:21.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.308+0000 7fddb659c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fddb8063d70 0x7fddb8160510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:21.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.308+0000 7fddb659c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fddb8063d70 0x7fddb8160510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56092/0 (socket says 172.21.15.88:56092) 2023-10-18T16:08:21.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.308+0000 7fddb659c700 1 -- 172.21.15.88:0/1729271589 learned_addr learned my addr 172.21.15.88:0/1729271589 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:08:21.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.309+0000 7fddb6d9d700 1 --2- 172.21.15.88:0/1729271589 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fddb8153b50 0x7fddb8152f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:21.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.309+0000 7fddb659c700 1 -- 172.21.15.88:0/1729271589 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fddb80c6a60 msgr2=0x7fddb814f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:21.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.309+0000 7fddb659c700 1 --2- 172.21.15.88:0/1729271589 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fddb80c6a60 0x7fddb814f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-10-18T16:08:21.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.309+0000 7fddb659c700 1 -- 172.21.15.88:0/1729271589 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fddb8153b50 msgr2=0x7fddb8152f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:21.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.309+0000 7fddb659c700 1 --2- 172.21.15.88:0/1729271589 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fddb8153b50 0x7fddb8152f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:21.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.309+0000 7fddb659c700 1 -- 172.21.15.88:0/1729271589 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddac009370 con 0x7fddb8063d70 2023-10-18T16:08:21.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.309+0000 7fddb6d9d700 1 --2- 172.21.15.88:0/1729271589 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fddb8153b50 0x7fddb8152f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-10-18T16:08:21.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.309+0000 7fddb659c700 1 --2- 172.21.15.88:0/1729271589 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fddb8063d70 0x7fddb8160510 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fdda8009f20 tx=0x7fdda800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:08:21.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.309+0000 7fdda67fc700 1 -- 172.21.15.88:0/1729271589 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda800cc80 con 0x7fddb8063d70 2023-10-18T16:08:21.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.309+0000 7fdda67fc700 1 -- 172.21.15.88:0/1729271589 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdda800f040 con 0x7fddb8063d70 2023-10-18T16:08:21.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.309+0000 7fddb759e700 1 -- 172.21.15.88:0/1729271589 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddb80ba810 con 0x7fddb8063d70 2023-10-18T16:08:21.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.309+0000 7fdda67fc700 1 -- 172.21.15.88:0/1729271589 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda8013400 con 0x7fddb8063d70 2023-10-18T16:08:21.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.309+0000 7fddb759e700 1 -- 172.21.15.88:0/1729271589 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddb80baf60 con 0x7fddb8063d70 2023-10-18T16:08:21.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.310+0000 7fddb759e700 1 -- 172.21.15.88:0/1729271589 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddb8084340 con 0x7fddb8063d70 2023-10-18T16:08:21.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.310+0000 7fdda67fc700 1 -- 172.21.15.88:0/1729271589 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7fdda8013770 con 0x7fddb8063d70 2023-10-18T16:08:21.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.310+0000 7fdda67fc700 1 --2- 172.21.15.88:0/1729271589 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fdd9805f150 0x7fdd98061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:21.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.311+0000 7fdda67fc700 1 -- 172.21.15.88:0/1729271589 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6121+0+0 (secure 0 0 0) 0x7fdda807aaf0 con 0x7fddb8063d70 2023-10-18T16:08:21.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.311+0000 7fddb5d9b700 1 --2- 172.21.15.88:0/1729271589 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fdd9805f150 0x7fdd98061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:21.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.311+0000 7fddb5d9b700 1 --2- 172.21.15.88:0/1729271589 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fdd9805f150 0x7fdd98061610 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fddb814ffa0 tx=0x7fddac017000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:08:21.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.315+0000 7fdda67fc700 1 -- 172.21.15.88:0/1729271589 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7fdda804bdc0 con 0x7fddb8063d70 2023-10-18T16:08:21.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.519+0000 7fddb759e700 1 -- 172.21.15.88:0/1729271589 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fddb814c7e0 con 0x7fddb8063d70 2023-10-18T16:08:21.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.519+0000 7fdda67fc700 1 -- 172.21.15.88:0/1729271589 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7fdda804f3e0 con 0x7fddb8063d70 2023-10-18T16:08:21.523 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:08:21.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.521+0000 7fddb759e700 1 -- 172.21.15.88:0/1729271589 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fdd9805f150 msgr2=0x7fdd98061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:21.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.521+0000 7fddb759e700 1 --2- 172.21.15.88:0/1729271589 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fdd9805f150 0x7fdd98061610 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fddb814ffa0 tx=0x7fddac017000).stop 2023-10-18T16:08:21.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.521+0000 7fddb759e700 1 -- 172.21.15.88:0/1729271589 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fddb8063d70 msgr2=0x7fddb8160510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:21.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.521+0000 7fddb759e700 1 --2- 172.21.15.88:0/1729271589 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fddb8063d70 0x7fddb8160510 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fdda8009f20 tx=0x7fdda800b040).stop 2023-10-18T16:08:21.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.521+0000 7fddb759e700 1 -- 172.21.15.88:0/1729271589 shutdown_connections 2023-10-18T16:08:21.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.521+0000 7fddb759e700 1 --2- 172.21.15.88:0/1729271589 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fdd9805f150 0x7fdd98061610 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:21.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.521+0000 7fddb759e700 1 --2- 172.21.15.88:0/1729271589 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fddb8153b50 0x7fddb8152f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:21.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.521+0000 7fddb759e700 1 --2- 172.21.15.88:0/1729271589 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fddb8063d70 0x7fddb8160510 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:21.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.521+0000 7fddb759e700 1 --2- 172.21.15.88:0/1729271589 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fddb80c6a60 0x7fddb814f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-10-18T16:08:21.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.521+0000 7fddb759e700 1 -- 172.21.15.88:0/1729271589 >> 172.21.15.88:0/1729271589 conn(0x7fddb805e050 msgr2=0x7fddb80be410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:08:21.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.521+0000 7fddb759e700 1 -- 172.21.15.88:0/1729271589 shutdown_connections 2023-10-18T16:08:21.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:21.521+0000 7fddb759e700 1 -- 172.21.15.88:0/1729271589 wait complete. 2023-10-18T16:08:21.536 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:08:23.647 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~34s 2023-10-18T16:08:24.538 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:08:24.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.660+0000 7fbd7bdbe700 1 Processor -- start 2023-10-18T16:08:24.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.660+0000 7fbd7bdbe700 1 -- start start 2023-10-18T16:08:24.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.661+0000 7fbd7bdbe700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbd7414d8c0 0x7fbd74064440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:24.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.661+0000 7fbd7bdbe700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbd74065360 0x7fbd74064930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:24.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.661+0000 7fbd7bdbe700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd74064150 con 0x7fbd740a9050 2023-10-18T16:08:24.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.661+0000 7fbd7bdbe700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd74153a40 con 0x7fbd7414d8c0 2023-10-18T16:08:24.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.661+0000 7fbd7bdbe700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd74063e70 con 0x7fbd74065360 2023-10-18T16:08:24.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.661+0000 7fbd79b5a700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbd7414d8c0 0x7fbd74064440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:24.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.661+0000 7fbd79b5a700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbd7414d8c0 0x7fbd74064440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56104/0 (socket says 172.21.15.88:56104) 2023-10-18T16:08:24.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.661+0000 7fbd79359700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fbd740a9050 0x7fbd74153ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:50308/0 (socket says 172.21.15.88:50308) 2023-10-18T16:08:24.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.661+0000 7fbd79b5a700 1 -- 172.21.15.88:0/2524503763 learned_addr learned my addr 172.21.15.88:0/2524503763 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:08:24.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.661+0000 7fbd79b5a700 1 -- 172.21.15.88:0/2524503763 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbd74065360 msgr2=0x7fbd74064930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:08:24.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.661+0000 7fbd7a35b700 1 --2- 172.21.15.88:0/2524503763 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbd74065360 0x7fbd74064930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:24.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.661+0000 7fbd79b5a700 1 --2- 172.21.15.88:0/2524503763 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbd74065360 0x7fbd74064930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:24.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.661+0000 7fbd78b58700 1 -- 172.21.15.88:0/2524503763 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2430306664 0 0) 0x7fbd74064150 con 0x7fbd740a9050 2023-10-18T16:08:24.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.661+0000 7fbd79b5a700 1 -- 172.21.15.88:0/2524503763 >> v1:172.21.15.88:6789/0 conn(0x7fbd740a9050 legacy=0x7fbd74153ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:24.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.661+0000 7fbd79b5a700 1 -- 172.21.15.88:0/2524503763 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd740c72c0 con 0x7fbd7414d8c0 2023-10-18T16:08:24.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.662+0000 7fbd79b5a700 1 --2- 172.21.15.88:0/2524503763 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbd7414d8c0 0x7fbd74064440 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fbd70006fd0 tx=0x7fbd700090f0).ready entity=mon.0 client_cookie=6d88753f7d731b74 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:08:24.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.662+0000 7fbd78b58700 1 -- 172.21.15.88:0/2524503763 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd7000b600 con 0x7fbd7414d8c0 2023-10-18T16:08:24.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.662+0000 7fbd78b58700 1 -- 172.21.15.88:0/2524503763 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd7000b760 con 0x7fbd7414d8c0 2023-10-18T16:08:24.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.662+0000 7fbd78b58700 1 -- 172.21.15.88:0/2524503763 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd7000bb00 con 0x7fbd7414d8c0 2023-10-18T16:08:24.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.662+0000 7fbd7bdbe700 1 -- 172.21.15.88:0/2524503763 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbd7414d8c0 msgr2=0x7fbd74064440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:24.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.662+0000 7fbd7bdbe700 1 --2- 172.21.15.88:0/2524503763 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbd7414d8c0 0x7fbd74064440 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fbd70006fd0 tx=0x7fbd700090f0).stop 2023-10-18T16:08:24.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.662+0000 7fbd7bdbe700 1 -- 172.21.15.88:0/2524503763 shutdown_connections 2023-10-18T16:08:24.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.662+0000 7fbd7bdbe700 1 --2- 172.21.15.88:0/2524503763 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbd74065360 0x7fbd74064930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:24.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.662+0000 7fbd7bdbe700 1 --2- 172.21.15.88:0/2524503763 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbd7414d8c0 0x7fbd74064440 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:24.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.662+0000 7fbd7bdbe700 1 -- 172.21.15.88:0/2524503763 >> 172.21.15.88:0/2524503763 conn(0x7fbd7406a810 msgr2=0x7fbd740c6a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:08:24.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.662+0000 7fbd7bdbe700 1 -- 172.21.15.88:0/2524503763 shutdown_connections 2023-10-18T16:08:24.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.662+0000 7fbd7bdbe700 1 -- 172.21.15.88:0/2524503763 wait complete. 2023-10-18T16:08:24.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.663+0000 7fbd7bdbe700 1 Processor -- start 2023-10-18T16:08:24.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.663+0000 7fbd7bdbe700 1 -- start start 2023-10-18T16:08:24.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.664+0000 7fbd7bdbe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbd74065360 0x7fbd741643b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:24.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.664+0000 7fbd7bdbe700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbd740a9050 0x7fbd740bcba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:24.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.664+0000 7fbd7bdbe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd7414d8c0 0x7fbd740aa180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:24.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.664+0000 7fbd7bdbe700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd741648a0 con 0x7fbd74065360 2023-10-18T16:08:24.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.664+0000 7fbd7bdbe700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd740a9a10 con 0x7fbd7414d8c0 2023-10-18T16:08:24.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.664+0000 7fbd7bdbe700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd740a9cf0 con 0x7fbd740a9050 2023-10-18T16:08:24.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.664+0000 7fbd79b5a700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbd74065360 0x7fbd741643b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:24.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.664+0000 7fbd79b5a700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbd74065360 0x7fbd741643b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56106/0 (socket says 172.21.15.88:56106) 2023-10-18T16:08:24.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.664+0000 7fbd79b5a700 1 -- 172.21.15.88:0/2436328044 learned_addr learned my addr 172.21.15.88:0/2436328044 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:08:24.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.664+0000 7fbd79359700 1 --2- 172.21.15.88:0/2436328044 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbd740a9050 0x7fbd740bcba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:24.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.664+0000 7fbd79b5a700 1 -- 172.21.15.88:0/2436328044 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbd740a9050 msgr2=0x7fbd740bcba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:24.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.665+0000 7fbd79b5a700 1 --2- 172.21.15.88:0/2436328044 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbd740a9050 0x7fbd740bcba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:24.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.665+0000 7fbd79b5a700 1 -- 172.21.15.88:0/2436328044 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd7414d8c0 msgr2=0x7fbd740aa180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:24.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.665+0000 7fbd79b5a700 1 --2- 172.21.15.88:0/2436328044 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd7414d8c0 0x7fbd740aa180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-10-18T16:08:24.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.665+0000 7fbd79b5a700 1 -- 172.21.15.88:0/2436328044 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd70009370 con 0x7fbd74065360 2023-10-18T16:08:24.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.665+0000 7fbd79b5a700 1 --2- 172.21.15.88:0/2436328044 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbd74065360 0x7fbd741643b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fbd70006b50 tx=0x7fbd70017000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:08:24.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.665+0000 7fbd69ffb700 1 -- 172.21.15.88:0/2436328044 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd70006910 con 0x7fbd74065360 2023-10-18T16:08:24.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.665+0000 7fbd69ffb700 1 -- 172.21.15.88:0/2436328044 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd7000b8d0 con 0x7fbd74065360 2023-10-18T16:08:24.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.665+0000 7fbd7bdbe700 1 -- 172.21.15.88:0/2436328044 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd740c31d0 con 0x7fbd74065360 2023-10-18T16:08:24.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.665+0000 7fbd69ffb700 1 -- 172.21.15.88:0/2436328044 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd70024470 con 0x7fbd74065360 2023-10-18T16:08:24.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.665+0000 7fbd7bdbe700 1 -- 172.21.15.88:0/2436328044 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd74150af0 con 0x7fbd74065360 2023-10-18T16:08:24.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.666+0000 7fbd7bdbe700 1 -- 172.21.15.88:0/2436328044 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd7406c4e0 con 0x7fbd74065360 2023-10-18T16:08:24.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.666+0000 7fbd69ffb700 1 -- 172.21.15.88:0/2436328044 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7fbd70021070 con 0x7fbd74065360 2023-10-18T16:08:24.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.667+0000 7fbd69ffb700 1 --2- 172.21.15.88:0/2436328044 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fbd5805f120 0x7fbd580615e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:24.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.667+0000 7fbd69ffb700 1 -- 172.21.15.88:0/2436328044 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6121+0+0 (secure 0 0 0) 0x7fbd7007bc40 con 0x7fbd74065360 2023-10-18T16:08:24.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.667+0000 7fbd79359700 1 --2- 172.21.15.88:0/2436328044 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fbd5805f120 0x7fbd580615e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:24.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.667+0000 7fbd79359700 1 --2- 172.21.15.88:0/2436328044 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fbd5805f120 0x7fbd580615e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fbd64007fd0 tx=0x7fbd64009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:08:24.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.671+0000 7fbd69ffb700 1 -- 172.21.15.88:0/2436328044 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7fbd7004cf10 con 0x7fbd74065360 2023-10-18T16:08:24.876 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.873+0000 7fbd7bdbe700 1 -- 172.21.15.88:0/2436328044 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fbd740c75a0 con 0x7fbd74065360 2023-10-18T16:08:24.876 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.873+0000 7fbd69ffb700 1 -- 172.21.15.88:0/2436328044 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7fbd70050530 con 0x7fbd74065360 2023-10-18T16:08:24.877 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:08:24.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.875+0000 7fbd7bdbe700 1 -- 172.21.15.88:0/2436328044 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fbd5805f120 msgr2=0x7fbd580615e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:24.879 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.875+0000 7fbd7bdbe700 1 --2- 172.21.15.88:0/2436328044 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fbd5805f120 0x7fbd580615e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fbd64007fd0 tx=0x7fbd64009040).stop 2023-10-18T16:08:24.879 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.875+0000 7fbd7bdbe700 1 -- 172.21.15.88:0/2436328044 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbd74065360 msgr2=0x7fbd741643b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:24.879 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.875+0000 7fbd7bdbe700 1 --2- 172.21.15.88:0/2436328044 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbd74065360 0x7fbd741643b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fbd70006b50 tx=0x7fbd70017000).stop 2023-10-18T16:08:24.879 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.875+0000 7fbd7bdbe700 1 -- 172.21.15.88:0/2436328044 shutdown_connections 2023-10-18T16:08:24.879 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.875+0000 7fbd7bdbe700 1 --2- 172.21.15.88:0/2436328044 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fbd5805f120 0x7fbd580615e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:24.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.875+0000 7fbd7bdbe700 1 --2- 172.21.15.88:0/2436328044 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd7414d8c0 0x7fbd740aa180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-10-18T16:08:24.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.875+0000 7fbd7bdbe700 1 --2- 172.21.15.88:0/2436328044 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbd74065360 0x7fbd741643b0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:24.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.875+0000 7fbd7bdbe700 1 --2- 172.21.15.88:0/2436328044 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbd740a9050 0x7fbd740bcba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:24.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.875+0000 7fbd7bdbe700 1 -- 172.21.15.88:0/2436328044 >> 172.21.15.88:0/2436328044 conn(0x7fbd7406a810 msgr2=0x7fbd74153ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:08:24.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.875+0000 7fbd7bdbe700 1 -- 172.21.15.88:0/2436328044 shutdown_connections 2023-10-18T16:08:24.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:24.875+0000 7fbd7bdbe700 1 -- 172.21.15.88:0/2436328044 wait complete. 2023-10-18T16:08:24.890 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:08:27.891 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:08:28.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.012+0000 7fea68b3c700 1 Processor -- start 2023-10-18T16:08:28.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.012+0000 7fea68b3c700 1 -- start start 2023-10-18T16:08:28.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.013+0000 7fea68b3c700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fea64062ae0 0x7fea6406a140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:28.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.013+0000 7fea68b3c700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea640c36e0 con 0x7fea64062650 2023-10-18T16:08:28.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.013+0000 7fea68b3c700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea64067ba0 con 0x7fea64069d60 2023-10-18T16:08:28.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.013+0000 7fea68b3c700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea64067e60 con 0x7fea64062ae0 2023-10-18T16:08:28.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.013+0000 7fea62d9d700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fea64062650 0x7fea6414d2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:39090/0 (socket says 172.21.15.88:39090) 2023-10-18T16:08:28.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.013+0000 7fea62d9d700 1 -- 172.21.15.88:0/3189722918 learned_addr learned my addr 172.21.15.88:0/3189722918 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:08:28.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.013+0000 7fea59d9b700 1 --2- 172.21.15.88:0/3189722918 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fea64062ae0 0x7fea6406a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:28.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.013+0000 7fea61d9b700 1 -- 172.21.15.88:0/3189722918 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4030372579 0 0) 0x7fea640c36e0 con 0x7fea64062650 2023-10-18T16:08:28.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.014+0000 7fea61d9b700 1 -- 172.21.15.88:0/3189722918 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea500033f0 con 0x7fea64062650 2023-10-18T16:08:28.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.014+0000 7fea61d9b700 1 -- 172.21.15.88:0/3189722918 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 29579715 0 0) 0x7fea500033f0 con 0x7fea64062650 2023-10-18T16:08:28.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.014+0000 7fea61d9b700 1 -- 172.21.15.88:0/3189722918 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fea64062ae0 msgr2=0x7fea6406a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:08:28.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.014+0000 7fea61d9b700 1 --2- 172.21.15.88:0/3189722918 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fea64062ae0 0x7fea6406a140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:28.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.014+0000 7fea61d9b700 1 -- 172.21.15.88:0/3189722918 >> v1:172.21.15.151:6789/0 conn(0x7fea64069d60 legacy=0x7fea640c3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:28.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.014+0000 7fea61d9b700 1 -- 172.21.15.88:0/3189722918 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea64068140 con 0x7fea64062650 2023-10-18T16:08:28.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.014+0000 7fea61d9b700 1 -- 172.21.15.88:0/3189722918 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441715075 0 0) 0x7fea5c0017f0 con 0x7fea64062650 2023-10-18T16:08:28.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.014+0000 7fea61d9b700 1 -- 172.21.15.88:0/3189722918 >> v1:172.21.15.88:6789/0 conn(0x7fea64062650 legacy=0x7fea6414d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:28.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.014+0000 7fea61d9b700 1 --2- 172.21.15.88:0/3189722918 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fea64062ae0 0x7fea50003910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:28.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.014+0000 7fea61d9b700 1 --2- 172.21.15.88:0/3189722918 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fea50003e40 0x7fea50004690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:28.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.014+0000 7fea61d9b700 1 --2- 172.21.15.88:0/3189722918 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fea50004b80 0x7fea50005000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:28.029 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.014+0000 7fea61d9b700 1 -- 172.21.15.88:0/3189722918 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea64067e60 con 0x7fea50003e40 2023-10-18T16:08:28.029 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.014+0000 7fea61d9b700 1 -- 172.21.15.88:0/3189722918 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea5000b990 con 0x7fea50004b80 2023-10-18T16:08:28.029 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.014+0000 7fea61d9b700 1 -- 172.21.15.88:0/3189722918 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea5000bb50 con 0x7fea64062ae0 2023-10-18T16:08:28.029 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.014+0000 7fea59d9b700 1 --2- 172.21.15.88:0/3189722918 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fea50003e40 0x7fea50004690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:28.030 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.014+0000 7fea6259c700 1 --2- 172.21.15.88:0/3189722918 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fea64062ae0 0x7fea50003910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:28.030 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.014+0000 7fea62d9d700 1 --2- 172.21.15.88:0/3189722918 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fea50004b80 0x7fea50005000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:28.030 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.014+0000 7fea59d9b700 1 -- 172.21.15.88:0/3189722918 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fea64062ae0 msgr2=0x7fea50003910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:08:28.030 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.014+0000 7fea59d9b700 1 --2- 172.21.15.88:0/3189722918 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fea64062ae0 0x7fea50003910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:28.030 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.014+0000 7fea59d9b700 1 -- 172.21.15.88:0/3189722918 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fea50004b80 msgr2=0x7fea50005000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:08:28.031 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.014+0000 7fea59d9b700 1 --2- 172.21.15.88:0/3189722918 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fea50004b80 0x7fea50005000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:28.031 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.014+0000 7fea59d9b700 1 -- 172.21.15.88:0/3189722918 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea500033f0 con 0x7fea50003e40 2023-10-18T16:08:28.031 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.014+0000 7fea59d9b700 1 --2- 172.21.15.88:0/3189722918 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fea50003e40 0x7fea50004690 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fea4c009a60 tx=0x7fea4c009060).ready entity=mon.0 client_cookie=f0ad750fd50ecfa3 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:08:28.031 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.014+0000 7fea61d9b700 1 -- 172.21.15.88:0/3189722918 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea4c009ea0 con 0x7fea50003e40 2023-10-18T16:08:28.031 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.014+0000 7fea61d9b700 1 -- 172.21.15.88:0/3189722918 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea4c009430 con 0x7fea50003e40 2023-10-18T16:08:28.032 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.014+0000 7fea61d9b700 1 -- 172.21.15.88:0/3189722918 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea4c00e870 con 0x7fea50003e40 2023-10-18T16:08:28.032 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.015+0000 7fea68b3c700 1 -- 172.21.15.88:0/3189722918 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fea50003e40 msgr2=0x7fea50004690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:28.032 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.015+0000 7fea68b3c700 1 --2- 172.21.15.88:0/3189722918 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fea50003e40 0x7fea50004690 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fea4c009a60 tx=0x7fea4c009060).stop 2023-10-18T16:08:28.032 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.015+0000 7fea62d9d700 1 -- 172.21.15.88:0/3189722918 reap_dead start 2023-10-18T16:08:28.032 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.015+0000 7fea68b3c700 1 -- 172.21.15.88:0/3189722918 shutdown_connections 2023-10-18T16:08:28.033 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.015+0000 7fea68b3c700 1 -- 172.21.15.88:0/3189722918 >> 172.21.15.88:0/3189722918 conn(0x7fea64065b10 msgr2=0x7fea64065f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:08:28.033 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.015+0000 7fea68b3c700 1 -- 172.21.15.88:0/3189722918 shutdown_connections 2023-10-18T16:08:28.033 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.015+0000 7fea68b3c700 1 -- 172.21.15.88:0/3189722918 wait complete. 2023-10-18T16:08:28.033 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.016+0000 7fea68b3c700 1 Processor -- start 2023-10-18T16:08:28.033 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.016+0000 7fea68b3c700 1 -- start start 2023-10-18T16:08:28.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.017+0000 7fea68b3c700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fea50003e40 0x7fea641616e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:28.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.017+0000 7fea68b3c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fea64146020 0x7fea6414f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:28.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.017+0000 7fea68b3c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fea6414f7b0 0x7fea640b80f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:28.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.017+0000 7fea68b3c700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea64161bd0 con 0x7fea6414f7b0 2023-10-18T16:08:28.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.017+0000 7fea68b3c700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea641516d0 con 0x7fea64146020 2023-10-18T16:08:28.035 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.017+0000 7fea68b3c700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea64152780 con 0x7fea50003e40 2023-10-18T16:08:28.035 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.017+0000 7fea62d9d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fea6414f7b0 0x7fea640b80f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:28.035 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.017+0000 7fea62d9d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fea6414f7b0 0x7fea640b80f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40932/0 (socket says 172.21.15.88:40932) 2023-10-18T16:08:28.035 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.017+0000 7fea62d9d700 1 -- 172.21.15.88:0/3296046954 learned_addr learned my addr 172.21.15.88:0/3296046954 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:08:28.035 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.017+0000 7fea6259c700 1 --2- 172.21.15.88:0/3296046954 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fea50003e40 0x7fea641616e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:28.036 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.018+0000 7fea59d9b700 1 --2- 172.21.15.88:0/3296046954 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fea64146020 0x7fea6414f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:28.036 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.018+0000 7fea62d9d700 1 -- 172.21.15.88:0/3296046954 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fea50003e40 msgr2=0x7fea641616e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:28.036 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.018+0000 7fea62d9d700 1 --2- 172.21.15.88:0/3296046954 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fea50003e40 0x7fea641616e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:28.036 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.018+0000 7fea62d9d700 1 -- 172.21.15.88:0/3296046954 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fea64146020 msgr2=0x7fea6414f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:28.036 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.018+0000 7fea62d9d700 1 --2- 172.21.15.88:0/3296046954 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fea64146020 0x7fea6414f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:28.037 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.018+0000 7fea62d9d700 1 -- 172.21.15.88:0/3296046954 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea64151920 con 0x7fea6414f7b0 2023-10-18T16:08:28.037 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.018+0000 7fea6259c700 1 --2- 172.21.15.88:0/3296046954 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fea50003e40 0x7fea641616e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-10-18T16:08:28.037 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.018+0000 7fea59d9b700 1 --2- 172.21.15.88:0/3296046954 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fea64146020 0x7fea6414f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-10-18T16:08:28.037 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.018+0000 7fea62d9d700 1 --2- 172.21.15.88:0/3296046954 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fea6414f7b0 0x7fea640b80f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7fea5c000f80 tx=0x7fea5c001390).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:08:28.037 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.018+0000 7fea5affd700 1 -- 172.21.15.88:0/3296046954 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea5c00e900 con 0x7fea6414f7b0 2023-10-18T16:08:28.038 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.018+0000 7fea5affd700 1 -- 172.21.15.88:0/3296046954 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea5c00ea80 con 0x7fea6414f7b0 2023-10-18T16:08:28.038 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.018+0000 7fea68b3c700 1 -- 172.21.15.88:0/3296046954 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea64151c00 con 0x7fea6414f7b0 2023-10-18T16:08:28.038 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.018+0000 7fea5affd700 1 -- 172.21.15.88:0/3296046954 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea5c00ee40 con 0x7fea6414f7b0 2023-10-18T16:08:28.038 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.018+0000 7fea68b3c700 1 -- 172.21.15.88:0/3296046954 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea641521c0 con 0x7fea6414f7b0 2023-10-18T16:08:28.038 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.019+0000 7fea68b3c700 1 -- 172.21.15.88:0/3296046954 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea6406c4e0 con 0x7fea6414f7b0 2023-10-18T16:08:28.039 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.019+0000 7fea5affd700 1 -- 172.21.15.88:0/3296046954 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7fea5c014640 con 0x7fea6414f7b0 2023-10-18T16:08:28.039 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.019+0000 7fea5affd700 1 --2- 172.21.15.88:0/3296046954 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fea4405f180 0x7fea44061640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:28.039 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.020+0000 7fea5affd700 1 -- 172.21.15.88:0/3296046954 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6121+0+0 (secure 0 0 0) 0x7fea5c07a0d0 con 0x7fea6414f7b0 2023-10-18T16:08:28.039 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.020+0000 7fea6259c700 1 --2- 172.21.15.88:0/3296046954 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fea4405f180 0x7fea44061640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:28.039 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.020+0000 7fea6259c700 1 --2- 172.21.15.88:0/3296046954 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fea4405f180 0x7fea44061640 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fea54007a40 tx=0x7fea54009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:08:28.040 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.024+0000 7fea5affd700 1 -- 172.21.15.88:0/3296046954 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7fea5c049ac0 con 0x7fea6414f7b0 2023-10-18T16:08:28.228 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.224+0000 7fea68b3c700 1 -- 172.21.15.88:0/3296046954 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fea64068420 con 0x7fea6414f7b0 2023-10-18T16:08:28.228 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.224+0000 7fea5affd700 1 -- 172.21.15.88:0/3296046954 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7fea5c04b3a0 con 0x7fea6414f7b0 2023-10-18T16:08:28.228 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:08:28.230 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.226+0000 7fea68b3c700 1 -- 172.21.15.88:0/3296046954 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fea4405f180 msgr2=0x7fea44061640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:28.230 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.226+0000 7fea68b3c700 1 --2- 172.21.15.88:0/3296046954 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fea4405f180 0x7fea44061640 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fea54007a40 tx=0x7fea54009040).stop 2023-10-18T16:08:28.230 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.226+0000 7fea68b3c700 1 -- 172.21.15.88:0/3296046954 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fea6414f7b0 msgr2=0x7fea640b80f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:28.230 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.226+0000 7fea68b3c700 1 --2- 172.21.15.88:0/3296046954 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fea6414f7b0 0x7fea640b80f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7fea5c000f80 tx=0x7fea5c001390).stop 2023-10-18T16:08:28.231 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.226+0000 7fea68b3c700 1 -- 172.21.15.88:0/3296046954 shutdown_connections 2023-10-18T16:08:28.231 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.226+0000 7fea68b3c700 1 --2- 172.21.15.88:0/3296046954 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fea4405f180 0x7fea44061640 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:28.231 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.226+0000 7fea68b3c700 1 --2- 172.21.15.88:0/3296046954 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fea6414f7b0 0x7fea640b80f0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:28.231 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.226+0000 7fea68b3c700 1 --2- 172.21.15.88:0/3296046954 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fea50003e40 0x7fea641616e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:28.232 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.226+0000 7fea68b3c700 1 --2- 172.21.15.88:0/3296046954 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fea64146020 0x7fea6414f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:28.232 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.226+0000 7fea68b3c700 1 -- 172.21.15.88:0/3296046954 >> 172.21.15.88:0/3296046954 conn(0x7fea64065b10 msgr2=0x7fea641506a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:08:28.232 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.226+0000 7fea68b3c700 1 -- 172.21.15.88:0/3296046954 shutdown_connections 2023-10-18T16:08:28.232 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:28.226+0000 7fea68b3c700 1 -- 172.21.15.88:0/3296046954 wait complete. 2023-10-18T16:08:28.242 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:08:30.461 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~41s 2023-10-18T16:08:31.243 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:08:31.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.363+0000 7fc066fa8700 1 Processor -- start 2023-10-18T16:08:31.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.364+0000 7fc066fa8700 1 -- start start 2023-10-18T16:08:31.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.364+0000 7fc066fa8700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc0601533f0 0x7fc060151ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:31.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.364+0000 7fc066fa8700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fc0601523d0 0x7fc0600c36c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:31.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.364+0000 7fc066fa8700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0600a9670 con 0x7fc060153010 2023-10-18T16:08:31.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.364+0000 7fc066fa8700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc06014df50 con 0x7fc0601523d0 2023-10-18T16:08:31.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.364+0000 7fc066fa8700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc060151a50 con 0x7fc0601533f0 2023-10-18T16:08:31.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.364+0000 7fc064d44700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fc060153010 0x7fc060064d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:39098/0 (socket says 172.21.15.88:39098) 2023-10-18T16:08:31.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.364+0000 7fc064d44700 1 -- 172.21.15.88:0/3345062466 learned_addr learned my addr 172.21.15.88:0/3345062466 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:08:31.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.365+0000 7fc0577fe700 1 -- 172.21.15.88:0/3345062466 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3533307918 0 0) 0x7fc0600a9670 con 0x7fc060153010 2023-10-18T16:08:31.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.365+0000 7fc0577fe700 1 -- 172.21.15.88:0/3345062466 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0480033f0 con 0x7fc060153010 2023-10-18T16:08:31.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.365+0000 7fc0577fe700 1 -- 172.21.15.88:0/3345062466 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4015808056 0 0) 0x7fc0480033f0 con 0x7fc060153010 2023-10-18T16:08:31.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.365+0000 7fc0577fe700 1 -- 172.21.15.88:0/3345062466 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc0601533f0 msgr2=0x7fc060151ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:08:31.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.365+0000 7fc0577fe700 1 --2- 172.21.15.88:0/3345062466 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc0601533f0 0x7fc060151ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:08:31.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.365+0000 7fc0577fe700 1 -- 172.21.15.88:0/3345062466 >> v2:172.21.15.151:3300/0 conn(0x7fc0601523d0 msgr2=0x7fc0600c36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:08:31.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.365+0000 7fc0577fe700 1 --2- 172.21.15.88:0/3345062466 >> v2:172.21.15.151:3300/0 conn(0x7fc0601523d0 0x7fc0600c36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:08:31.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.365+0000 7fc0577fe700 1 -- 172.21.15.88:0/3345062466 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0600a8af0 con 0x7fc060153010 2023-10-18T16:08:31.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.365+0000 7fc0577fe700 1 -- 172.21.15.88:0/3345062466 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441715075 0 0) 0x7fc05c001470 con 0x7fc060153010 2023-10-18T16:08:31.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.365+0000 7fc0577fe700 1 -- 172.21.15.88:0/3345062466 >> v1:172.21.15.88:6789/0 conn(0x7fc060153010 legacy=0x7fc060064d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:31.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.365+0000 7fc0577fe700 1 --2- 172.21.15.88:0/3345062466 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc0480038e0 0x7fc048003d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:31.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.365+0000 7fc0577fe700 1 --2- 172.21.15.88:0/3345062466 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc0601533f0 0x7fc048004690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:31.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.365+0000 7fc0577fe700 1 --2- 172.21.15.88:0/3345062466 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc048004ca0 0x7fc048005120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:31.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.365+0000 7fc0577fe700 1 -- 172.21.15.88:0/3345062466 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc06014df50 con 0x7fc048004ca0 2023-10-18T16:08:31.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.365+0000 7fc0577fe700 1 -- 172.21.15.88:0/3345062466 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc060151a50 con 0x7fc0480038e0 2023-10-18T16:08:31.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.365+0000 7fc0577fe700 1 -- 172.21.15.88:0/3345062466 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc04800bbc0 con 0x7fc0601533f0 2023-10-18T16:08:31.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.365+0000 7fc065545700 1 --2- 172.21.15.88:0/3345062466 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc048004ca0 0x7fc048005120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:31.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.365+0000 7fc064d44700 1 --2- 172.21.15.88:0/3345062466 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc0480038e0 0x7fc048003d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:31.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.365+0000 7fc057fff700 1 --2- 172.21.15.88:0/3345062466 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc0601533f0 0x7fc048004690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:31.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.365+0000 7fc065545700 1 -- 172.21.15.88:0/3345062466 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc0601533f0 msgr2=0x7fc048004690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:08:31.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.365+0000 7fc065545700 1 --2- 172.21.15.88:0/3345062466 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc0601533f0 0x7fc048004690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:31.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.365+0000 7fc065545700 1 -- 172.21.15.88:0/3345062466 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc0480038e0 msgr2=0x7fc048003d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:08:31.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.365+0000 7fc065545700 1 --2- 172.21.15.88:0/3345062466 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc0480038e0 0x7fc048003d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:31.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.365+0000 7fc065545700 1 -- 172.21.15.88:0/3345062466 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0480033f0 con 0x7fc048004ca0 2023-10-18T16:08:31.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.366+0000 7fc065545700 1 --2- 172.21.15.88:0/3345062466 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc048004ca0 0x7fc048005120 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7fc058009fa0 tx=0x7fc0580095f0).ready entity=mon.0 client_cookie=6582f0d559184cf0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:08:31.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.366+0000 7fc0577fe700 1 -- 172.21.15.88:0/3345062466 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0580098e0 con 0x7fc048004ca0 2023-10-18T16:08:31.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.366+0000 7fc0577fe700 1 -- 172.21.15.88:0/3345062466 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc05800bc90 con 0x7fc048004ca0 2023-10-18T16:08:31.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.366+0000 7fc0577fe700 1 -- 172.21.15.88:0/3345062466 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0580156c0 con 0x7fc048004ca0 2023-10-18T16:08:31.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.366+0000 7fc066fa8700 1 -- 172.21.15.88:0/3345062466 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc048004ca0 msgr2=0x7fc048005120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:31.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.366+0000 7fc066fa8700 1 --2- 172.21.15.88:0/3345062466 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc048004ca0 0x7fc048005120 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7fc058009fa0 tx=0x7fc0580095f0).stop 2023-10-18T16:08:31.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.366+0000 7fc065545700 1 -- 172.21.15.88:0/3345062466 reap_dead start 2023-10-18T16:08:31.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.366+0000 7fc066fa8700 1 -- 172.21.15.88:0/3345062466 shutdown_connections 2023-10-18T16:08:31.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.366+0000 7fc066fa8700 1 -- 172.21.15.88:0/3345062466 >> 172.21.15.88:0/3345062466 conn(0x7fc060069d80 msgr2=0x7fc06006a190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:08:31.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.366+0000 7fc066fa8700 1 -- 172.21.15.88:0/3345062466 shutdown_connections 2023-10-18T16:08:31.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.366+0000 7fc066fa8700 1 -- 172.21.15.88:0/3345062466 wait complete. 2023-10-18T16:08:31.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.367+0000 7fc066fa8700 1 Processor -- start 2023-10-18T16:08:31.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.368+0000 7fc066fa8700 1 -- start start 2023-10-18T16:08:31.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.368+0000 7fc066fa8700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc048004ca0 0x7fc0600bf9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:31.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.368+0000 7fc066fa8700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc0600c07a0 0x7fc0601539b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:31.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.368+0000 7fc066fa8700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc060153ea0 0x7fc0601618c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:31.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.368+0000 7fc066fa8700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc06015ffe0 con 0x7fc048004ca0 2023-10-18T16:08:31.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.368+0000 7fc066fa8700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc060161430 con 0x7fc060153ea0 2023-10-18T16:08:31.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.368+0000 7fc066fa8700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0601601f0 con 0x7fc0600c07a0 2023-10-18T16:08:31.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.368+0000 7fc064d44700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc048004ca0 0x7fc0600bf9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:31.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.368+0000 7fc064d44700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc048004ca0 0x7fc0600bf9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40944/0 (socket says 172.21.15.88:40944) 2023-10-18T16:08:31.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.368+0000 7fc064d44700 1 -- 172.21.15.88:0/2971379763 learned_addr learned my addr 172.21.15.88:0/2971379763 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:08:31.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.368+0000 7fc057fff700 1 --2- 172.21.15.88:0/2971379763 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc0600c07a0 0x7fc0601539b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:31.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.369+0000 7fc065545700 1 --2- 172.21.15.88:0/2971379763 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc060153ea0 0x7fc0601618c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:31.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.369+0000 7fc064d44700 1 -- 172.21.15.88:0/2971379763 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc0600c07a0 msgr2=0x7fc0601539b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:31.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.369+0000 7fc064d44700 1 --2- 172.21.15.88:0/2971379763 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc0600c07a0 0x7fc0601539b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:31.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.369+0000 7fc064d44700 1 -- 172.21.15.88:0/2971379763 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc060153ea0 msgr2=0x7fc0601618c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:31.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.369+0000 7fc064d44700 1 --2- 172.21.15.88:0/2971379763 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc060153ea0 0x7fc0601618c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:31.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.369+0000 7fc064d44700 1 -- 172.21.15.88:0/2971379763 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0601604d0 con 0x7fc048004ca0 2023-10-18T16:08:31.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.369+0000 7fc057fff700 1 --2- 172.21.15.88:0/2971379763 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc0600c07a0 0x7fc0601539b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:08:31.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.369+0000 7fc065545700 1 --2- 172.21.15.88:0/2971379763 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc060153ea0 0x7fc0601618c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:08:31.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.369+0000 7fc064d44700 1 --2- 172.21.15.88:0/2971379763 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc048004ca0 0x7fc0600bf9a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fc05c000fd0 tx=0x7fc05c001e90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:08:31.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.369+0000 7fc054ff9700 1 -- 172.21.15.88:0/2971379763 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc05c011590 con 0x7fc048004ca0 2023-10-18T16:08:31.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.369+0000 7fc054ff9700 1 -- 172.21.15.88:0/2971379763 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc05c0116f0 con 0x7fc048004ca0 2023-10-18T16:08:31.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.369+0000 7fc066fa8700 1 -- 172.21.15.88:0/2971379763 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0601607b0 con 0x7fc048004ca0 2023-10-18T16:08:31.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.369+0000 7fc054ff9700 1 -- 172.21.15.88:0/2971379763 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc05c011a70 con 0x7fc048004ca0 2023-10-18T16:08:31.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.369+0000 7fc066fa8700 1 -- 172.21.15.88:0/2971379763 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc060160d70 con 0x7fc048004ca0 2023-10-18T16:08:31.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.370+0000 7fc066fa8700 1 -- 172.21.15.88:0/2971379763 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0600a8890 con 0x7fc048004ca0 2023-10-18T16:08:31.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.370+0000 7fc054ff9700 1 -- 172.21.15.88:0/2971379763 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7fc05c01d3b0 con 0x7fc048004ca0 2023-10-18T16:08:31.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.371+0000 7fc054ff9700 1 --2- 172.21.15.88:0/2971379763 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fc04c05f150 0x7fc04c061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:31.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.371+0000 7fc054ff9700 1 -- 172.21.15.88:0/2971379763 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6295+0+0 (secure 0 0 0) 0x7fc05c079bc0 con 0x7fc048004ca0 2023-10-18T16:08:31.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.371+0000 7fc057fff700 1 --2- 172.21.15.88:0/2971379763 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fc04c05f150 0x7fc04c061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:31.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.371+0000 7fc057fff700 1 --2- 172.21.15.88:0/2971379763 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fc04c05f150 0x7fc04c061610 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fc050006ba0 tx=0x7fc050008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:08:31.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.375+0000 7fc054ff9700 1 -- 172.21.15.88:0/2971379763 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7fc05c04ae30 con 0x7fc048004ca0 2023-10-18T16:08:31.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.576+0000 7fc066fa8700 1 -- 172.21.15.88:0/2971379763 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc0601615a0 con 0x7fc048004ca0 2023-10-18T16:08:31.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.577+0000 7fc054ff9700 1 -- 172.21.15.88:0/2971379763 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7fc05c019030 con 0x7fc048004ca0 2023-10-18T16:08:31.581 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:08:31.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.579+0000 7fc066fa8700 1 -- 172.21.15.88:0/2971379763 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fc04c05f150 msgr2=0x7fc04c061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:31.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.579+0000 7fc066fa8700 1 --2- 172.21.15.88:0/2971379763 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fc04c05f150 0x7fc04c061610 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fc050006ba0 tx=0x7fc050008040).stop 2023-10-18T16:08:31.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.579+0000 7fc066fa8700 1 -- 172.21.15.88:0/2971379763 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc048004ca0 msgr2=0x7fc0600bf9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:31.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.579+0000 7fc066fa8700 1 --2- 172.21.15.88:0/2971379763 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc048004ca0 0x7fc0600bf9a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fc05c000fd0 tx=0x7fc05c001e90).stop 2023-10-18T16:08:31.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.579+0000 7fc066fa8700 1 -- 172.21.15.88:0/2971379763 shutdown_connections 2023-10-18T16:08:31.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.579+0000 7fc066fa8700 1 --2- 172.21.15.88:0/2971379763 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fc04c05f150 0x7fc04c061610 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:31.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.579+0000 7fc066fa8700 1 --2- 172.21.15.88:0/2971379763 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc060153ea0 0x7fc0601618c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:31.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.579+0000 7fc066fa8700 1 --2- 172.21.15.88:0/2971379763 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc048004ca0 0x7fc0600bf9a0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:31.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.579+0000 7fc066fa8700 1 --2- 172.21.15.88:0/2971379763 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc0600c07a0 0x7fc0601539b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:31.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.579+0000 7fc066fa8700 1 -- 172.21.15.88:0/2971379763 >> 172.21.15.88:0/2971379763 conn(0x7fc060069d80 msgr2=0x7fc0600a9cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:08:31.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.579+0000 7fc066fa8700 1 -- 172.21.15.88:0/2971379763 shutdown_connections 2023-10-18T16:08:31.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:31.579+0000 7fc066fa8700 1 -- 172.21.15.88:0/2971379763 wait complete. 2023-10-18T16:08:31.594 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:08:33.427 INFO:tasks.ceph.mds.f.smithi151.stderr:terminate called after throwing an instance of 'boost::wrapexcept' 2023-10-18T16:08:33.427 INFO:tasks.ceph.mds.f.smithi151.stderr: what(): boost::bad_get: failed value get using boost::get 2023-10-18T16:08:33.427 INFO:tasks.ceph.mds.f.smithi151.stderr:*** Caught signal (Aborted) ** 2023-10-18T16:08:33.428 INFO:tasks.ceph.mds.f.smithi151.stderr: in thread 7f760ea65700 thread_name:ms_dispatch 2023-10-18T16:08:33.428 INFO:tasks.ceph.mds.f.smithi151.stderr: ceph version 16.2.14-100-gb4ddd52c (b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5) pacific (stable) 2023-10-18T16:08:33.429 INFO:tasks.ceph.mds.f.smithi151.stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f7616268b20] 2023-10-18T16:08:33.429 INFO:tasks.ceph.mds.f.smithi151.stderr: 2: gsignal() 2023-10-18T16:08:33.429 INFO:tasks.ceph.mds.f.smithi151.stderr: 3: abort() 2023-10-18T16:08:33.429 INFO:tasks.ceph.mds.f.smithi151.stderr: 4: /lib64/libstdc++.so.6(+0x9009b) [0x7f761566909b] 2023-10-18T16:08:33.429 INFO:tasks.ceph.mds.f.smithi151.stderr: 5: /lib64/libstdc++.so.6(+0x9653c) [0x7f761566f53c] 2023-10-18T16:08:33.430 INFO:tasks.ceph.mds.f.smithi151.stderr: 6: /lib64/libstdc++.so.6(+0x96597) [0x7f761566f597] 2023-10-18T16:08:33.430 INFO:tasks.ceph.mds.f.smithi151.stderr: 7: /lib64/libstdc++.so.6(+0x967f8) [0x7f761566f7f8] 2023-10-18T16:08:33.430 INFO:tasks.ceph.mds.f.smithi151.stderr: 8: ceph-mds(+0x212b43) [0x56232ccc6b43] 2023-10-18T16:08:33.430 INFO:tasks.ceph.mds.f.smithi151.stderr: 9: (Option::size_t const md_config_t::get_val(ConfigValues const&, std::basic_string_view >) const+0x5c) [0x56232cd3b89c] 2023-10-18T16:08:33.431 INFO:tasks.ceph.mds.f.smithi151.stderr: 10: (SessionMap::SessionMap(MDSRank*)+0x2c2) [0x56232cf836a2] 2023-10-18T16:08:33.431 INFO:tasks.ceph.mds.f.smithi151.stderr: 11: (MDSRank::MDSRank(int, std::__cxx11::basic_string, std::allocator >, ceph::fair_mutex&, std::shared_ptr&, CommonSafeTimer&, Beacon&, std::unique_ptr >&, Messenger*, MonClient*, MgrClient*, Context*, Context*, boost::asio::io_context&)+0x214) [0x56232cca0724] 2023-10-18T16:08:33.431 INFO:tasks.ceph.mds.f.smithi151.stderr: 12: (MDSRankDispatcher::MDSRankDispatcher(int, std::__cxx11::basic_string, std::allocator >, ceph::fair_mutex&, std::shared_ptr&, CommonSafeTimer&, Beacon&, std::unique_ptr >&, Messenger*, MonClient*, MgrClient*, Context*, Context*, boost::asio::io_context&)+0x101) [0x56232cca2181] 2023-10-18T16:08:33.431 INFO:tasks.ceph.mds.f.smithi151.stderr: 13: (MDSDaemon::handle_mds_map(boost::intrusive_ptr const&)+0x1608) [0x56232cc72348] 2023-10-18T16:08:33.431 INFO:tasks.ceph.mds.f.smithi151.stderr: 14: (MDSDaemon::handle_core_message(boost::intrusive_ptr const&)+0xcd) [0x56232cc7544d] 2023-10-18T16:08:33.431 INFO:tasks.ceph.mds.f.smithi151.stderr: 15: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0xc3) [0x56232cc75fa3] 2023-10-18T16:08:33.432 INFO:tasks.ceph.mds.f.smithi151.stderr: 16: (DispatchQueue::entry()+0x126a) [0x7f76174c24ea] 2023-10-18T16:08:33.432 INFO:tasks.ceph.mds.f.smithi151.stderr: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7617576d51] 2023-10-18T16:08:33.432 INFO:tasks.ceph.mds.f.smithi151.stderr: 18: /lib64/libpthread.so.0(+0x814a) [0x7f761625e14a] 2023-10-18T16:08:33.432 INFO:tasks.ceph.mds.f.smithi151.stderr: 19: clone() 2023-10-18T16:08:33.432 INFO:tasks.ceph.mds.f.smithi151.stderr:2023-10-18T16:08:33.426+0000 7f760ea65700 -1 *** Caught signal (Aborted) ** 2023-10-18T16:08:33.433 INFO:tasks.ceph.mds.f.smithi151.stderr: in thread 7f760ea65700 thread_name:ms_dispatch 2023-10-18T16:08:33.433 INFO:tasks.ceph.mds.f.smithi151.stderr: 2023-10-18T16:08:33.433 INFO:tasks.ceph.mds.f.smithi151.stderr: ceph version 16.2.14-100-gb4ddd52c (b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5) pacific (stable) 2023-10-18T16:08:33.433 INFO:tasks.ceph.mds.f.smithi151.stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f7616268b20] 2023-10-18T16:08:33.433 INFO:tasks.ceph.mds.f.smithi151.stderr: 2: gsignal() 2023-10-18T16:08:33.434 INFO:tasks.ceph.mds.f.smithi151.stderr: 3: abort() 2023-10-18T16:08:33.434 INFO:tasks.ceph.mds.f.smithi151.stderr: 4: /lib64/libstdc++.so.6(+0x9009b) [0x7f761566909b] 2023-10-18T16:08:33.434 INFO:tasks.ceph.mds.f.smithi151.stderr: 5: /lib64/libstdc++.so.6(+0x9653c) [0x7f761566f53c] 2023-10-18T16:08:33.434 INFO:tasks.ceph.mds.f.smithi151.stderr: 6: /lib64/libstdc++.so.6(+0x96597) [0x7f761566f597] 2023-10-18T16:08:33.435 INFO:tasks.ceph.mds.f.smithi151.stderr: 7: /lib64/libstdc++.so.6(+0x967f8) [0x7f761566f7f8] 2023-10-18T16:08:33.435 INFO:tasks.ceph.mds.f.smithi151.stderr: 8: ceph-mds(+0x212b43) [0x56232ccc6b43] 2023-10-18T16:08:33.435 INFO:tasks.ceph.mds.f.smithi151.stderr: 9: (Option::size_t const md_config_t::get_val(ConfigValues const&, std::basic_string_view >) const+0x5c) [0x56232cd3b89c] 2023-10-18T16:08:33.435 INFO:tasks.ceph.mds.f.smithi151.stderr: 10: (SessionMap::SessionMap(MDSRank*)+0x2c2) [0x56232cf836a2] 2023-10-18T16:08:33.436 INFO:tasks.ceph.mds.f.smithi151.stderr: 11: (MDSRank::MDSRank(int, std::__cxx11::basic_string, std::allocator >, ceph::fair_mutex&, std::shared_ptr&, CommonSafeTimer&, Beacon&, std::unique_ptr >&, Messenger*, MonClient*, MgrClient*, Context*, Context*, boost::asio::io_context&)+0x214) [0x56232cca0724] 2023-10-18T16:08:33.436 INFO:tasks.ceph.mds.f.smithi151.stderr: 12: (MDSRankDispatcher::MDSRankDispatcher(int, std::__cxx11::basic_string, std::allocator >, ceph::fair_mutex&, std::shared_ptr&, CommonSafeTimer&, Beacon&, std::unique_ptr >&, Messenger*, MonClient*, MgrClient*, Context*, Context*, boost::asio::io_context&)+0x101) [0x56232cca2181] 2023-10-18T16:08:33.436 INFO:tasks.ceph.mds.f.smithi151.stderr: 13: (MDSDaemon::handle_mds_map(boost::intrusive_ptr const&)+0x1608) [0x56232cc72348] 2023-10-18T16:08:33.436 INFO:tasks.ceph.mds.f.smithi151.stderr: 14: (MDSDaemon::handle_core_message(boost::intrusive_ptr const&)+0xcd) [0x56232cc7544d] 2023-10-18T16:08:33.436 INFO:tasks.ceph.mds.f.smithi151.stderr: 15: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0xc3) [0x56232cc75fa3] 2023-10-18T16:08:33.437 INFO:tasks.ceph.mds.f.smithi151.stderr: 16: (DispatchQueue::entry()+0x126a) [0x7f76174c24ea] 2023-10-18T16:08:33.437 INFO:tasks.ceph.mds.f.smithi151.stderr: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7617576d51] 2023-10-18T16:08:33.437 INFO:tasks.ceph.mds.f.smithi151.stderr: 18: /lib64/libpthread.so.0(+0x814a) [0x7f761625e14a] 2023-10-18T16:08:33.437 INFO:tasks.ceph.mds.f.smithi151.stderr: 19: clone() 2023-10-18T16:08:33.437 INFO:tasks.ceph.mds.f.smithi151.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-10-18T16:08:33.437 INFO:tasks.ceph.mds.f.smithi151.stderr: 2023-10-18T16:08:33.438 INFO:tasks.ceph.mds.f.smithi151.stderr: 0> 2023-10-18T16:08:33.426+0000 7f760ea65700 -1 *** Caught signal (Aborted) ** 2023-10-18T16:08:33.438 INFO:tasks.ceph.mds.f.smithi151.stderr: in thread 7f760ea65700 thread_name:ms_dispatch 2023-10-18T16:08:33.439 INFO:tasks.ceph.mds.f.smithi151.stderr: 2023-10-18T16:08:33.439 INFO:tasks.ceph.mds.f.smithi151.stderr: ceph version 16.2.14-100-gb4ddd52c (b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5) pacific (stable) 2023-10-18T16:08:33.439 INFO:tasks.ceph.mds.f.smithi151.stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f7616268b20] 2023-10-18T16:08:33.439 INFO:tasks.ceph.mds.f.smithi151.stderr: 2: gsignal() 2023-10-18T16:08:33.440 INFO:tasks.ceph.mds.f.smithi151.stderr: 3: abort() 2023-10-18T16:08:33.440 INFO:tasks.ceph.mds.f.smithi151.stderr: 4: /lib64/libstdc++.so.6(+0x9009b) [0x7f761566909b] 2023-10-18T16:08:33.440 INFO:tasks.ceph.mds.f.smithi151.stderr: 5: /lib64/libstdc++.so.6(+0x9653c) [0x7f761566f53c] 2023-10-18T16:08:33.440 INFO:tasks.ceph.mds.f.smithi151.stderr: 6: /lib64/libstdc++.so.6(+0x96597) [0x7f761566f597] 2023-10-18T16:08:33.441 INFO:tasks.ceph.mds.f.smithi151.stderr: 7: /lib64/libstdc++.so.6(+0x967f8) [0x7f761566f7f8] 2023-10-18T16:08:33.441 INFO:tasks.ceph.mds.f.smithi151.stderr: 8: ceph-mds(+0x212b43) [0x56232ccc6b43] 2023-10-18T16:08:33.441 INFO:tasks.ceph.mds.f.smithi151.stderr: 9: (Option::size_t const md_config_t::get_val(ConfigValues const&, std::basic_string_view >) const+0x5c) [0x56232cd3b89c] 2023-10-18T16:08:33.441 INFO:tasks.ceph.mds.f.smithi151.stderr: 10: (SessionMap::SessionMap(MDSRank*)+0x2c2) [0x56232cf836a2] 2023-10-18T16:08:33.441 INFO:tasks.ceph.mds.f.smithi151.stderr: 11: (MDSRank::MDSRank(int, std::__cxx11::basic_string, std::allocator >, ceph::fair_mutex&, std::shared_ptr&, CommonSafeTimer&, Beacon&, std::unique_ptr >&, Messenger*, MonClient*, MgrClient*, Context*, Context*, boost::asio::io_context&)+0x214) [0x56232cca0724] 2023-10-18T16:08:33.442 INFO:tasks.ceph.mds.f.smithi151.stderr: 12: (MDSRankDispatcher::MDSRankDispatcher(int, std::__cxx11::basic_string, std::allocator >, ceph::fair_mutex&, std::shared_ptr&, CommonSafeTimer&, Beacon&, std::unique_ptr >&, Messenger*, MonClient*, MgrClient*, Context*, Context*, boost::asio::io_context&)+0x101) [0x56232cca2181] 2023-10-18T16:08:33.442 INFO:tasks.ceph.mds.f.smithi151.stderr: 13: (MDSDaemon::handle_mds_map(boost::intrusive_ptr const&)+0x1608) [0x56232cc72348] 2023-10-18T16:08:33.442 INFO:tasks.ceph.mds.f.smithi151.stderr: 14: (MDSDaemon::handle_core_message(boost::intrusive_ptr const&)+0xcd) [0x56232cc7544d] 2023-10-18T16:08:33.442 INFO:tasks.ceph.mds.f.smithi151.stderr: 15: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0xc3) [0x56232cc75fa3] 2023-10-18T16:08:33.442 INFO:tasks.ceph.mds.f.smithi151.stderr: 16: (DispatchQueue::entry()+0x126a) [0x7f76174c24ea] 2023-10-18T16:08:33.443 INFO:tasks.ceph.mds.f.smithi151.stderr: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7617576d51] 2023-10-18T16:08:33.443 INFO:tasks.ceph.mds.f.smithi151.stderr: 18: /lib64/libpthread.so.0(+0x814a) [0x7f761625e14a] 2023-10-18T16:08:33.443 INFO:tasks.ceph.mds.f.smithi151.stderr: 19: clone() 2023-10-18T16:08:33.443 INFO:tasks.ceph.mds.f.smithi151.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-10-18T16:08:33.443 INFO:tasks.ceph.mds.f.smithi151.stderr: 2023-10-18T16:08:33.443 INFO:tasks.ceph.mds.f.smithi151.stderr: 0> 2023-10-18T16:08:33.426+0000 7f760ea65700 -1 *** Caught signal (Aborted) ** 2023-10-18T16:08:33.444 INFO:tasks.ceph.mds.f.smithi151.stderr: in thread 7f760ea65700 thread_name:ms_dispatch 2023-10-18T16:08:33.444 INFO:tasks.ceph.mds.f.smithi151.stderr: 2023-10-18T16:08:33.444 INFO:tasks.ceph.mds.f.smithi151.stderr: ceph version 16.2.14-100-gb4ddd52c (b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5) pacific (stable) 2023-10-18T16:08:33.444 INFO:tasks.ceph.mds.f.smithi151.stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f7616268b20] 2023-10-18T16:08:33.444 INFO:tasks.ceph.mds.f.smithi151.stderr: 2: gsignal() 2023-10-18T16:08:33.445 INFO:tasks.ceph.mds.f.smithi151.stderr: 3: abort() 2023-10-18T16:08:33.445 INFO:tasks.ceph.mds.f.smithi151.stderr: 4: /lib64/libstdc++.so.6(+0x9009b) [0x7f761566909b] 2023-10-18T16:08:33.445 INFO:tasks.ceph.mds.f.smithi151.stderr: 5: /lib64/libstdc++.so.6(+0x9653c) [0x7f761566f53c] 2023-10-18T16:08:33.445 INFO:tasks.ceph.mds.f.smithi151.stderr: 6: /lib64/libstdc++.so.6(+0x96597) [0x7f761566f597] 2023-10-18T16:08:33.446 INFO:tasks.ceph.mds.f.smithi151.stderr: 7: /lib64/libstdc++.so.6(+0x967f8) [0x7f761566f7f8] 2023-10-18T16:08:33.446 INFO:tasks.ceph.mds.f.smithi151.stderr: 8: ceph-mds(+0x212b43) [0x56232ccc6b43] 2023-10-18T16:08:33.446 INFO:tasks.ceph.mds.f.smithi151.stderr: 9: (Option::size_t const md_config_t::get_val(ConfigValues const&, std::basic_string_view >) const+0x5c) [0x56232cd3b89c] 2023-10-18T16:08:33.446 INFO:tasks.ceph.mds.f.smithi151.stderr: 10: (SessionMap::SessionMap(MDSRank*)+0x2c2) [0x56232cf836a2] 2023-10-18T16:08:33.447 INFO:tasks.ceph.mds.f.smithi151.stderr: 11: (MDSRank::MDSRank(int, std::__cxx11::basic_string, std::allocator >, ceph::fair_mutex&, std::shared_ptr&, CommonSafeTimer&, Beacon&, std::unique_ptr >&, Messenger*, MonClient*, MgrClient*, Context*, Context*, boost::asio::io_context&)+0x214) [0x56232cca0724] 2023-10-18T16:08:33.447 INFO:tasks.ceph.mds.f.smithi151.stderr: 12: (MDSRankDispatcher::MDSRankDispatcher(int, std::__cxx11::basic_string, std::allocator >, ceph::fair_mutex&, std::shared_ptr&, CommonSafeTimer&, Beacon&, std::unique_ptr >&, Messenger*, MonClient*, MgrClient*, Context*, Context*, boost::asio::io_context&)+0x101) [0x56232cca2181] 2023-10-18T16:08:33.447 INFO:tasks.ceph.mds.f.smithi151.stderr: 13: (MDSDaemon::handle_mds_map(boost::intrusive_ptr const&)+0x1608) [0x56232cc72348] 2023-10-18T16:08:33.447 INFO:tasks.ceph.mds.f.smithi151.stderr: 14: (MDSDaemon::handle_core_message(boost::intrusive_ptr const&)+0xcd) [0x56232cc7544d] 2023-10-18T16:08:33.447 INFO:tasks.ceph.mds.f.smithi151.stderr: 15: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0xc3) [0x56232cc75fa3] 2023-10-18T16:08:33.448 INFO:tasks.ceph.mds.f.smithi151.stderr: 16: (DispatchQueue::entry()+0x126a) [0x7f76174c24ea] 2023-10-18T16:08:33.448 INFO:tasks.ceph.mds.f.smithi151.stderr: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7617576d51] 2023-10-18T16:08:33.448 INFO:tasks.ceph.mds.f.smithi151.stderr: 18: /lib64/libpthread.so.0(+0x814a) [0x7f761625e14a] 2023-10-18T16:08:33.448 INFO:tasks.ceph.mds.f.smithi151.stderr: 19: clone() 2023-10-18T16:08:33.448 INFO:tasks.ceph.mds.f.smithi151.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-10-18T16:08:33.449 INFO:tasks.ceph.mds.f.smithi151.stderr: 2023-10-18T16:08:33.713 INFO:tasks.ceph.mds.f.smithi151.stderr:daemon-helper: command crashed with signal 6 2023-10-18T16:08:34.595 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:08:34.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.716+0000 7f540c838700 1 Processor -- start 2023-10-18T16:08:34.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.716+0000 7f540c838700 1 -- start start 2023-10-18T16:08:34.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.717+0000 7f540c838700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54080a9050 0x7f5408153ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:34.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.717+0000 7f540c838700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5408065360 0x7f54081543c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:34.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.717+0000 7f540c838700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5408064150 con 0x7f540814d8c0 2023-10-18T16:08:34.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.717+0000 7f540c838700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5408153a40 con 0x7f5408065360 2023-10-18T16:08:34.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.717+0000 7f540c838700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5408063e70 con 0x7f54080a9050 2023-10-18T16:08:34.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.717+0000 7f5406d9d700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5408065360 0x7f54081543c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:34.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.717+0000 7f5406d9d700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5408065360 0x7f54081543c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40950/0 (socket says 172.21.15.88:40950) 2023-10-18T16:08:34.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.717+0000 7f5406d9d700 1 -- 172.21.15.88:0/1884087592 learned_addr learned my addr 172.21.15.88:0/1884087592 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:08:34.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.718+0000 7f53fdd9b700 1 --2- 172.21.15.88:0/1884087592 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54080a9050 0x7f5408153ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:34.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.718+0000 7f5406d9d700 1 -- 172.21.15.88:0/1884087592 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54080a9050 msgr2=0x7f5408153ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:08:34.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.718+0000 7f5406d9d700 1 --2- 172.21.15.88:0/1884087592 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54080a9050 0x7f5408153ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:34.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.718+0000 7f5406d9d700 1 -- 172.21.15.88:0/1884087592 >> v1:172.21.15.151:6789/0 conn(0x7f540814d8c0 legacy=0x7f5408064440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:34.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.718+0000 7f5406d9d700 1 -- 172.21.15.88:0/1884087592 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54080c72c0 con 0x7f5408065360 2023-10-18T16:08:34.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.718+0000 7f5406d9d700 1 --2- 172.21.15.88:0/1884087592 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5408065360 0x7f54081543c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f5400009f20 tx=0x7f5400009840).ready entity=mon.0 client_cookie=96514e148e777528 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:08:34.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.718+0000 7f5405d9b700 1 -- 172.21.15.88:0/1884087592 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f540000e070 con 0x7f5408065360 2023-10-18T16:08:34.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.718+0000 7f5405d9b700 1 -- 172.21.15.88:0/1884087592 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f540000b9d0 con 0x7f5408065360 2023-10-18T16:08:34.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.718+0000 7f5405d9b700 1 -- 172.21.15.88:0/1884087592 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f540000bd50 con 0x7f5408065360 2023-10-18T16:08:34.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.719+0000 7f540c838700 1 -- 172.21.15.88:0/1884087592 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5408065360 msgr2=0x7f54081543c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:34.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.719+0000 7f540c838700 1 --2- 172.21.15.88:0/1884087592 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5408065360 0x7f54081543c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f5400009f20 tx=0x7f5400009840).stop 2023-10-18T16:08:34.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.719+0000 7f540c838700 1 -- 172.21.15.88:0/1884087592 shutdown_connections 2023-10-18T16:08:34.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.719+0000 7f540c838700 1 --2- 172.21.15.88:0/1884087592 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5408065360 0x7f54081543c0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:34.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.719+0000 7f540c838700 1 --2- 172.21.15.88:0/1884087592 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54080a9050 0x7f5408153ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:34.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.719+0000 7f540c838700 1 -- 172.21.15.88:0/1884087592 >> 172.21.15.88:0/1884087592 conn(0x7f540806a810 msgr2=0x7f54080c6a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:08:34.728 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.719+0000 7f540c838700 1 -- 172.21.15.88:0/1884087592 shutdown_connections 2023-10-18T16:08:34.728 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.719+0000 7f540c838700 1 -- 172.21.15.88:0/1884087592 wait complete. 2023-10-18T16:08:34.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.720+0000 7f540c838700 1 Processor -- start 2023-10-18T16:08:34.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.720+0000 7f540c838700 1 -- start start 2023-10-18T16:08:34.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.721+0000 7f540c838700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5408065360 0x7f54080c0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:34.732 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.721+0000 7f540c838700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54080a9050 0x7f5408163f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:34.732 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.721+0000 7f540c838700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f540814d8c0 0x7f5408164460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:34.732 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.721+0000 7f540c838700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54080aa1b0 con 0x7f5408065360 2023-10-18T16:08:34.732 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.721+0000 7f540c838700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54080aa5d0 con 0x7f540814d8c0 2023-10-18T16:08:34.732 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.721+0000 7f540c838700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5408164950 con 0x7f54080a9050 2023-10-18T16:08:34.733 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.721+0000 7f540659c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5408065360 0x7f54080c0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:34.733 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.721+0000 7f540659c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5408065360 0x7f54080c0bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40958/0 (socket says 172.21.15.88:40958) 2023-10-18T16:08:34.733 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.721+0000 7f540659c700 1 -- 172.21.15.88:0/994698062 learned_addr learned my addr 172.21.15.88:0/994698062 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:08:34.733 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.721+0000 7f53fdd9b700 1 --2- 172.21.15.88:0/994698062 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54080a9050 0x7f5408163f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:34.734 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.721+0000 7f5406d9d700 1 --2- 172.21.15.88:0/994698062 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f540814d8c0 0x7f5408164460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:34.734 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.721+0000 7f540659c700 1 -- 172.21.15.88:0/994698062 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54080a9050 msgr2=0x7f5408163f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:34.734 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.721+0000 7f540659c700 1 --2- 172.21.15.88:0/994698062 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54080a9050 0x7f5408163f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:34.734 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.721+0000 7f540659c700 1 -- 172.21.15.88:0/994698062 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f540814d8c0 msgr2=0x7f5408164460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:34.735 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.721+0000 7f540659c700 1 --2- 172.21.15.88:0/994698062 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f540814d8c0 0x7f5408164460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:34.735 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.721+0000 7f540659c700 1 -- 172.21.15.88:0/994698062 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5400009ae0 con 0x7f5408065360 2023-10-18T16:08:34.735 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.721+0000 7f540659c700 1 --2- 172.21.15.88:0/994698062 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5408065360 0x7f54080c0bd0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f53f800c440 tx=0x7f53f8007db0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:08:34.736 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.721+0000 7f53feffd700 1 -- 172.21.15.88:0/994698062 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f800ce20 con 0x7f5408065360 2023-10-18T16:08:34.736 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.721+0000 7f53feffd700 1 -- 172.21.15.88:0/994698062 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53f8001e90 con 0x7f5408065360 2023-10-18T16:08:34.736 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.721+0000 7f540c838700 1 -- 172.21.15.88:0/994698062 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54080ac9b0 con 0x7f5408065360 2023-10-18T16:08:34.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.721+0000 7f53feffd700 1 -- 172.21.15.88:0/994698062 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f800f6a0 con 0x7f5408065360 2023-10-18T16:08:34.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.721+0000 7f540c838700 1 -- 172.21.15.88:0/994698062 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54080aaac0 con 0x7f5408065360 2023-10-18T16:08:34.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.722+0000 7f540c838700 1 -- 172.21.15.88:0/994698062 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f540806c4e0 con 0x7f5408065360 2023-10-18T16:08:34.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.723+0000 7f53feffd700 1 -- 172.21.15.88:0/994698062 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f53f800fa50 con 0x7f5408065360 2023-10-18T16:08:34.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.723+0000 7f53feffd700 1 --2- 172.21.15.88:0/994698062 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f53e805f120 0x7f53e80615e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:34.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.723+0000 7f53feffd700 1 -- 172.21.15.88:0/994698062 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6381+0+0 (secure 0 0 0) 0x7f53f8079600 con 0x7f5408065360 2023-10-18T16:08:34.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.723+0000 7f53fdd9b700 1 --2- 172.21.15.88:0/994698062 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f53e805f120 0x7f53e80615e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:34.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.724+0000 7f53fdd9b700 1 --2- 172.21.15.88:0/994698062 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f53e805f120 0x7f53e80615e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f54080630f0 tx=0x7f53f0009060).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:08:34.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.727+0000 7f53feffd700 1 -- 172.21.15.88:0/994698062 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f53f8047ac0 con 0x7f5408065360 2023-10-18T16:08:34.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.926+0000 7f540c838700 1 -- 172.21.15.88:0/994698062 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f54080abbe0 con 0x7f5408065360 2023-10-18T16:08:34.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.927+0000 7f53feffd700 1 -- 172.21.15.88:0/994698062 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7f53f804a7d0 con 0x7f5408065360 2023-10-18T16:08:34.931 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:08:34.932 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.929+0000 7f540c838700 1 -- 172.21.15.88:0/994698062 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f53e805f120 msgr2=0x7f53e80615e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:34.932 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.929+0000 7f540c838700 1 --2- 172.21.15.88:0/994698062 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f53e805f120 0x7f53e80615e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f54080630f0 tx=0x7f53f0009060).stop 2023-10-18T16:08:34.933 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.929+0000 7f540c838700 1 -- 172.21.15.88:0/994698062 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5408065360 msgr2=0x7f54080c0bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:34.933 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.929+0000 7f540c838700 1 --2- 172.21.15.88:0/994698062 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5408065360 0x7f54080c0bd0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f53f800c440 tx=0x7f53f8007db0).stop 2023-10-18T16:08:34.933 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.929+0000 7f540c838700 1 -- 172.21.15.88:0/994698062 shutdown_connections 2023-10-18T16:08:34.933 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.929+0000 7f540c838700 1 --2- 172.21.15.88:0/994698062 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f53e805f120 0x7f53e80615e0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:34.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.929+0000 7f540c838700 1 --2- 172.21.15.88:0/994698062 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f540814d8c0 0x7f5408164460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:34.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.929+0000 7f540c838700 1 --2- 172.21.15.88:0/994698062 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5408065360 0x7f54080c0bd0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:34.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.929+0000 7f540c838700 1 --2- 172.21.15.88:0/994698062 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54080a9050 0x7f5408163f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:34.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.929+0000 7f540c838700 1 -- 172.21.15.88:0/994698062 >> 172.21.15.88:0/994698062 conn(0x7f540806a810 msgr2=0x7f5408151810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:08:34.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.929+0000 7f540c838700 1 -- 172.21.15.88:0/994698062 shutdown_connections 2023-10-18T16:08:34.935 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:34.929+0000 7f540c838700 1 -- 172.21.15.88:0/994698062 wait complete. 2023-10-18T16:08:34.944 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:08:37.173 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~48s 2023-10-18T16:08:37.174 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.f is failed for ~0s 2023-10-18T16:08:37.945 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:08:38.069 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.066+0000 7fb8db237700 1 Processor -- start 2023-10-18T16:08:38.070 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.066+0000 7fb8db237700 1 -- start start 2023-10-18T16:08:38.073 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.067+0000 7fb8db237700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb8d414d8c0 0x7fb8d4064440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:38.073 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.067+0000 7fb8db237700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb8d40a9050 0x7fb8d4064930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:38.073 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.067+0000 7fb8db237700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fb8d4065360 0x7fb8d4153ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:38.073 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.067+0000 7fb8db237700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8d4064150 con 0x7fb8d414d8c0 2023-10-18T16:08:38.073 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.067+0000 7fb8db237700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8d4153a40 con 0x7fb8d4065360 2023-10-18T16:08:38.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.067+0000 7fb8db237700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8d4063e70 con 0x7fb8d40a9050 2023-10-18T16:08:38.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.067+0000 7fb8d8fd3700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb8d414d8c0 0x7fb8d4064440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:38.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.067+0000 7fb8d8fd3700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb8d414d8c0 0x7fb8d4064440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40456/0 (socket says 172.21.15.88:40456) 2023-10-18T16:08:38.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.067+0000 7fb8d8fd3700 1 -- 172.21.15.88:0/930785057 learned_addr learned my addr 172.21.15.88:0/930785057 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:08:38.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.067+0000 7fb8d3fff700 1 --2- 172.21.15.88:0/930785057 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb8d40a9050 0x7fb8d4064930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:38.075 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.067+0000 7fb8d97d4700 1 --2- 172.21.15.88:0/930785057 >> v2:172.21.15.151:3300/0 conn(0x7fb8d4065360 0x7fb8d4153ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:38.075 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.067+0000 7fb8d8fd3700 1 -- 172.21.15.88:0/930785057 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb8d40a9050 msgr2=0x7fb8d4064930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:08:38.075 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.067+0000 7fb8d8fd3700 1 --2- 172.21.15.88:0/930785057 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb8d40a9050 0x7fb8d4064930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:38.075 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.067+0000 7fb8d8fd3700 1 -- 172.21.15.88:0/930785057 >> v2:172.21.15.151:3300/0 conn(0x7fb8d4065360 msgr2=0x7fb8d4153ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:08:38.075 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.067+0000 7fb8d8fd3700 1 --2- 172.21.15.88:0/930785057 >> v2:172.21.15.151:3300/0 conn(0x7fb8d4065360 0x7fb8d4153ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:38.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.067+0000 7fb8d8fd3700 1 -- 172.21.15.88:0/930785057 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8d40c72c0 con 0x7fb8d414d8c0 2023-10-18T16:08:38.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.067+0000 7fb8d97d4700 1 --2- 172.21.15.88:0/930785057 >> v2:172.21.15.151:3300/0 conn(0x7fb8d4065360 0x7fb8d4153ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:08:38.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.067+0000 7fb8d8fd3700 1 --2- 172.21.15.88:0/930785057 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8d414d8c0 0x7fb8d4064440 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7fb8d40ba710 tx=0x7fb8c80090f0).ready entity=mon.0 client_cookie=8b53df8f31a726cb server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:08:38.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.067+0000 7fb8d37fe700 1 -- 172.21.15.88:0/930785057 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8c800b520 con 0x7fb8d414d8c0 2023-10-18T16:08:38.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.068+0000 7fb8d37fe700 1 -- 172.21.15.88:0/930785057 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8c800b680 con 0x7fb8d414d8c0 2023-10-18T16:08:38.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.068+0000 7fb8d37fe700 1 -- 172.21.15.88:0/930785057 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8c800ba20 con 0x7fb8d414d8c0 2023-10-18T16:08:38.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.068+0000 7fb8db237700 1 -- 172.21.15.88:0/930785057 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8d414d8c0 msgr2=0x7fb8d4064440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:38.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.068+0000 7fb8db237700 1 --2- 172.21.15.88:0/930785057 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8d414d8c0 0x7fb8d4064440 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7fb8d40ba710 tx=0x7fb8c80090f0).stop 2023-10-18T16:08:38.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.068+0000 7fb8db237700 1 -- 172.21.15.88:0/930785057 shutdown_connections 2023-10-18T16:08:38.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.068+0000 7fb8db237700 1 --2- 172.21.15.88:0/930785057 >> v2:172.21.15.151:3300/0 conn(0x7fb8d4065360 0x7fb8d4153ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:38.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.068+0000 7fb8db237700 1 --2- 172.21.15.88:0/930785057 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8d414d8c0 0x7fb8d4064440 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:38.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.068+0000 7fb8db237700 1 --2- 172.21.15.88:0/930785057 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb8d40a9050 0x7fb8d4064930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:38.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.068+0000 7fb8db237700 1 -- 172.21.15.88:0/930785057 >> 172.21.15.88:0/930785057 conn(0x7fb8d406a810 msgr2=0x7fb8d40c6a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:08:38.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.068+0000 7fb8db237700 1 -- 172.21.15.88:0/930785057 shutdown_connections 2023-10-18T16:08:38.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.068+0000 7fb8db237700 1 -- 172.21.15.88:0/930785057 wait complete. 2023-10-18T16:08:38.079 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.069+0000 7fb8db237700 1 Processor -- start 2023-10-18T16:08:38.079 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.069+0000 7fb8db237700 1 -- start start 2023-10-18T16:08:38.081 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.070+0000 7fb8db237700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8d4065360 0x7fb8d40bbc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:38.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.070+0000 7fb8db237700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb8d40a9050 0x7fb8d40bc180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:38.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.070+0000 7fb8db237700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb8d414d8c0 0x7fb8d4148bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:38.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.070+0000 7fb8db237700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8d40bb870 con 0x7fb8d4065360 2023-10-18T16:08:38.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.070+0000 7fb8db237700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8d40c6b60 con 0x7fb8d414d8c0 2023-10-18T16:08:38.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.070+0000 7fb8db237700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8d40ab570 con 0x7fb8d40a9050 2023-10-18T16:08:38.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.070+0000 7fb8d8fd3700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8d4065360 0x7fb8d40bbc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:38.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.070+0000 7fb8d8fd3700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8d4065360 0x7fb8d40bbc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40460/0 (socket says 172.21.15.88:40460) 2023-10-18T16:08:38.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.070+0000 7fb8d8fd3700 1 -- 172.21.15.88:0/2958366446 learned_addr learned my addr 172.21.15.88:0/2958366446 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:08:38.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.070+0000 7fb8d97d4700 1 --2- 172.21.15.88:0/2958366446 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb8d414d8c0 0x7fb8d4148bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:38.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.070+0000 7fb8d3fff700 1 --2- 172.21.15.88:0/2958366446 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb8d40a9050 0x7fb8d40bc180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:38.084 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.070+0000 7fb8d8fd3700 1 -- 172.21.15.88:0/2958366446 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb8d40a9050 msgr2=0x7fb8d40bc180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:38.084 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.070+0000 7fb8d8fd3700 1 --2- 172.21.15.88:0/2958366446 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb8d40a9050 0x7fb8d40bc180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:38.084 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.070+0000 7fb8d8fd3700 1 -- 172.21.15.88:0/2958366446 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb8d414d8c0 msgr2=0x7fb8d4148bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:38.084 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.070+0000 7fb8d8fd3700 1 --2- 172.21.15.88:0/2958366446 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb8d414d8c0 0x7fb8d4148bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:38.084 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.070+0000 7fb8d8fd3700 1 -- 172.21.15.88:0/2958366446 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8c8009370 con 0x7fb8d4065360 2023-10-18T16:08:38.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.070+0000 7fb8d8fd3700 1 --2- 172.21.15.88:0/2958366446 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8d4065360 0x7fb8d40bbc90 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fb8d4071c10 tx=0x7fb8c8017000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:08:38.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.071+0000 7fb8d0ff9700 1 -- 172.21.15.88:0/2958366446 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8c8000bc0 con 0x7fb8d4065360 2023-10-18T16:08:38.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.071+0000 7fb8d0ff9700 1 -- 172.21.15.88:0/2958366446 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8c800b7f0 con 0x7fb8d4065360 2023-10-18T16:08:38.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.071+0000 7fb8db237700 1 -- 172.21.15.88:0/2958366446 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8d40ab850 con 0x7fb8d4065360 2023-10-18T16:08:38.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.071+0000 7fb8d0ff9700 1 -- 172.21.15.88:0/2958366446 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8c8000d20 con 0x7fb8d4065360 2023-10-18T16:08:38.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.071+0000 7fb8db237700 1 -- 172.21.15.88:0/2958366446 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8d40abe10 con 0x7fb8d4065360 2023-10-18T16:08:38.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.072+0000 7fb8db237700 1 -- 172.21.15.88:0/2958366446 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8d40ab340 con 0x7fb8d4065360 2023-10-18T16:08:38.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.072+0000 7fb8d0ff9700 1 -- 172.21.15.88:0/2958366446 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7fb8c801c630 con 0x7fb8d4065360 2023-10-18T16:08:38.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.072+0000 7fb8d0ff9700 1 --2- 172.21.15.88:0/2958366446 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fb8b805f150 0x7fb8b8061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:38.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.072+0000 7fb8d0ff9700 1 -- 172.21.15.88:0/2958366446 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6381+0+0 (secure 0 0 0) 0x7fb8c807bd90 con 0x7fb8d4065360 2023-10-18T16:08:38.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.073+0000 7fb8d3fff700 1 --2- 172.21.15.88:0/2958366446 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fb8b805f150 0x7fb8b8061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:38.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.073+0000 7fb8d3fff700 1 --2- 172.21.15.88:0/2958366446 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fb8b805f150 0x7fb8b8061610 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fb8d40c3490 tx=0x7fb8c0008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:08:38.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.076+0000 7fb8d0ff9700 1 -- 172.21.15.88:0/2958366446 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7fb8c804cf60 con 0x7fb8d4065360 2023-10-18T16:08:38.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.275+0000 7fb8db237700 1 -- 172.21.15.88:0/2958366446 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb8d406c4e0 con 0x7fb8d4065360 2023-10-18T16:08:38.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.275+0000 7fb8d0ff9700 1 -- 172.21.15.88:0/2958366446 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7fb8c802a020 con 0x7fb8d4065360 2023-10-18T16:08:38.279 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:08:38.281 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.277+0000 7fb8db237700 1 -- 172.21.15.88:0/2958366446 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fb8b805f150 msgr2=0x7fb8b8061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:38.281 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.277+0000 7fb8db237700 1 --2- 172.21.15.88:0/2958366446 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fb8b805f150 0x7fb8b8061610 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fb8d40c3490 tx=0x7fb8c0008040).stop 2023-10-18T16:08:38.281 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.277+0000 7fb8db237700 1 -- 172.21.15.88:0/2958366446 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8d4065360 msgr2=0x7fb8d40bbc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:38.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.277+0000 7fb8db237700 1 --2- 172.21.15.88:0/2958366446 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8d4065360 0x7fb8d40bbc90 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fb8d4071c10 tx=0x7fb8c8017000).stop 2023-10-18T16:08:38.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.277+0000 7fb8db237700 1 -- 172.21.15.88:0/2958366446 shutdown_connections 2023-10-18T16:08:38.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.277+0000 7fb8db237700 1 --2- 172.21.15.88:0/2958366446 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fb8b805f150 0x7fb8b8061610 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:38.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.277+0000 7fb8db237700 1 --2- 172.21.15.88:0/2958366446 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb8d414d8c0 0x7fb8d4148bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:38.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.277+0000 7fb8db237700 1 --2- 172.21.15.88:0/2958366446 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8d4065360 0x7fb8d40bbc90 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:38.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.277+0000 7fb8db237700 1 --2- 172.21.15.88:0/2958366446 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb8d40a9050 0x7fb8d40bc180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:38.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.277+0000 7fb8db237700 1 -- 172.21.15.88:0/2958366446 >> 172.21.15.88:0/2958366446 conn(0x7fb8d406a810 msgr2=0x7fb8d4153ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:08:38.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.277+0000 7fb8db237700 1 -- 172.21.15.88:0/2958366446 shutdown_connections 2023-10-18T16:08:38.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:38.277+0000 7fb8db237700 1 -- 172.21.15.88:0/2958366446 wait complete. 2023-10-18T16:08:38.292 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:08:38.299 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-18T16:08:38.301 DEBUG:teuthology.orchestra.run.smithi151:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-18T16:08:41.294 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:08:41.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.413+0000 7f8a7ac72700 1 Processor -- start 2023-10-18T16:08:41.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.413+0000 7f8a7ac72700 1 -- start start 2023-10-18T16:08:41.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.413+0000 7f8a7ac72700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f8a740c08d0 0x7f8a740be0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:41.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.413+0000 7f8a7ac72700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a74153a40 con 0x7f8a74069b80 2023-10-18T16:08:41.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.413+0000 7f8a7ac72700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a74154000 con 0x7f8a740bd780 2023-10-18T16:08:41.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.413+0000 7f8a7ac72700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a7406a850 con 0x7f8a740c08d0 2023-10-18T16:08:41.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.414+0000 7f8a6bfff700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f8a74069b80 0x7f8a74149be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:59896/0 (socket says 172.21.15.88:59896) 2023-10-18T16:08:41.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.414+0000 7f8a6bfff700 1 -- 172.21.15.88:0/764508166 learned_addr learned my addr 172.21.15.88:0/764508166 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:08:41.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.414+0000 7f8a6b7fe700 1 -- 172.21.15.88:0/764508166 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2514920493 0 0) 0x7f8a74153a40 con 0x7f8a74069b80 2023-10-18T16:08:41.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.414+0000 7f8a6b7fe700 1 -- 172.21.15.88:0/764508166 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a5c0033f0 con 0x7f8a74069b80 2023-10-18T16:08:41.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.414+0000 7f8a7920f700 1 --2- 172.21.15.88:0/764508166 >> v2:172.21.15.151:3300/0 conn(0x7f8a740c08d0 0x7f8a740be0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:41.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.414+0000 7f8a6b7fe700 1 -- 172.21.15.88:0/764508166 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2790776064 0 0) 0x7f8a5c0033f0 con 0x7f8a74069b80 2023-10-18T16:08:41.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.414+0000 7f8a6b7fe700 1 -- 172.21.15.88:0/764508166 >> v2:172.21.15.151:3300/0 conn(0x7f8a740c08d0 msgr2=0x7f8a740be0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:08:41.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.414+0000 7f8a6b7fe700 1 --2- 172.21.15.88:0/764508166 >> v2:172.21.15.151:3300/0 conn(0x7f8a740c08d0 0x7f8a740be0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:41.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.414+0000 7f8a6b7fe700 1 -- 172.21.15.88:0/764508166 >> v1:172.21.15.151:6789/0 conn(0x7f8a740bd780 legacy=0x7f8a740b6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:41.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.414+0000 7f8a6b7fe700 1 -- 172.21.15.88:0/764508166 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a740bdb60 con 0x7f8a74069b80 2023-10-18T16:08:41.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.414+0000 7f8a7920f700 1 --2- 172.21.15.88:0/764508166 >> v2:172.21.15.151:3300/0 conn(0x7f8a740c08d0 0x7f8a740be0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:08:41.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.414+0000 7f8a6b7fe700 1 -- 172.21.15.88:0/764508166 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441715075 0 0) 0x7f8a64001470 con 0x7f8a74069b80 2023-10-18T16:08:41.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.414+0000 7f8a6b7fe700 1 -- 172.21.15.88:0/764508166 >> v1:172.21.15.88:6789/0 conn(0x7f8a74069b80 legacy=0x7f8a74149be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:41.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.414+0000 7f8a6b7fe700 1 --2- 172.21.15.88:0/764508166 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a5c0038e0 0x7f8a5c003d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:41.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.414+0000 7f8a6b7fe700 1 --2- 172.21.15.88:0/764508166 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8a5c004690 0x7f8a5c004b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:41.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.414+0000 7f8a6b7fe700 1 --2- 172.21.15.88:0/764508166 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8a5c005050 0x7f8a5c00b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:41.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.414+0000 7f8a6b7fe700 1 -- 172.21.15.88:0/764508166 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a7406a850 con 0x7f8a5c0038e0 2023-10-18T16:08:41.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.414+0000 7f8a6b7fe700 1 -- 172.21.15.88:0/764508166 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5c00bf30 con 0x7f8a5c004690 2023-10-18T16:08:41.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.414+0000 7f8a6b7fe700 1 -- 172.21.15.88:0/764508166 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5c00c0f0 con 0x7f8a5c005050 2023-10-18T16:08:41.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.414+0000 7f8a78a0e700 1 --2- 172.21.15.88:0/764508166 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a5c0038e0 0x7f8a5c003d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:41.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.414+0000 7f8a6bfff700 1 --2- 172.21.15.88:0/764508166 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8a5c004690 0x7f8a5c004b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:41.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.414+0000 7f8a7920f700 1 --2- 172.21.15.88:0/764508166 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8a5c005050 0x7f8a5c00b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:41.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.415+0000 7f8a78a0e700 1 -- 172.21.15.88:0/764508166 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8a5c005050 msgr2=0x7f8a5c00b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:08:41.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.415+0000 7f8a78a0e700 1 --2- 172.21.15.88:0/764508166 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8a5c005050 0x7f8a5c00b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:41.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.415+0000 7f8a78a0e700 1 -- 172.21.15.88:0/764508166 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8a5c004690 msgr2=0x7f8a5c004b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:08:41.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.415+0000 7f8a78a0e700 1 --2- 172.21.15.88:0/764508166 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8a5c004690 0x7f8a5c004b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:41.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.415+0000 7f8a78a0e700 1 -- 172.21.15.88:0/764508166 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a5c0033f0 con 0x7f8a5c0038e0 2023-10-18T16:08:41.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.415+0000 7f8a7920f700 1 --2- 172.21.15.88:0/764508166 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8a5c005050 0x7f8a5c00b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:08:41.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.415+0000 7f8a7920f700 1 -- 172.21.15.88:0/764508166 reap_dead start 2023-10-18T16:08:41.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.415+0000 7f8a78a0e700 1 --2- 172.21.15.88:0/764508166 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a5c0038e0 0x7f8a5c003d00 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f8a70001730 tx=0x7f8a70007bb0).ready entity=mon.0 client_cookie=88eecf7bc9368433 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:08:41.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.415+0000 7f8a6b7fe700 1 -- 172.21.15.88:0/764508166 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a70007020 con 0x7f8a5c0038e0 2023-10-18T16:08:41.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.415+0000 7f8a6b7fe700 1 -- 172.21.15.88:0/764508166 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a70007180 con 0x7f8a5c0038e0 2023-10-18T16:08:41.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.415+0000 7f8a6b7fe700 1 -- 172.21.15.88:0/764508166 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a70007520 con 0x7f8a5c0038e0 2023-10-18T16:08:41.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.415+0000 7f8a7ac72700 1 -- 172.21.15.88:0/764508166 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a5c0038e0 msgr2=0x7f8a5c003d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:41.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.415+0000 7f8a7ac72700 1 --2- 172.21.15.88:0/764508166 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a5c0038e0 0x7f8a5c003d00 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f8a70001730 tx=0x7f8a70007bb0).stop 2023-10-18T16:08:41.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.415+0000 7f8a7ac72700 1 -- 172.21.15.88:0/764508166 shutdown_connections 2023-10-18T16:08:41.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.415+0000 7f8a7ac72700 1 --2- 172.21.15.88:0/764508166 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a5c0038e0 0x7f8a5c003d00 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:41.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.415+0000 7f8a7ac72700 1 -- 172.21.15.88:0/764508166 >> 172.21.15.88:0/764508166 conn(0x7f8a74063760 msgr2=0x7f8a74063b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:08:41.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.415+0000 7f8a7ac72700 1 -- 172.21.15.88:0/764508166 shutdown_connections 2023-10-18T16:08:41.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.415+0000 7f8a7ac72700 1 -- 172.21.15.88:0/764508166 wait complete. 2023-10-18T16:08:41.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.416+0000 7f8a7ac72700 1 Processor -- start 2023-10-18T16:08:41.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.417+0000 7f8a7ac72700 1 -- start start 2023-10-18T16:08:41.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.417+0000 7f8a7ac72700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a5c0038e0 0x7f8a74161740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:41.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.417+0000 7f8a7ac72700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8a7414f800 0x7f8a7414fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:41.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.417+0000 7f8a7ac72700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8a7414ce60 0x7f8a7414d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:41.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.417+0000 7f8a7ac72700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a74161c30 con 0x7f8a5c0038e0 2023-10-18T16:08:41.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.417+0000 7f8a7ac72700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a7414e380 con 0x7f8a7414ce60 2023-10-18T16:08:41.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.417+0000 7f8a7ac72700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a7414edb0 con 0x7f8a7414f800 2023-10-18T16:08:41.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.418+0000 7f8a78a0e700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a5c0038e0 0x7f8a74161740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:41.434 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.418+0000 7f8a78a0e700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a5c0038e0 0x7f8a74161740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40470/0 (socket says 172.21.15.88:40470) 2023-10-18T16:08:41.434 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.418+0000 7f8a78a0e700 1 -- 172.21.15.88:0/3392923887 learned_addr learned my addr 172.21.15.88:0/3392923887 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:08:41.434 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.418+0000 7f8a6bfff700 1 --2- 172.21.15.88:0/3392923887 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8a7414f800 0x7f8a7414fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:41.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.418+0000 7f8a7920f700 1 --2- 172.21.15.88:0/3392923887 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8a7414ce60 0x7f8a7414d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:41.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.418+0000 7f8a78a0e700 1 -- 172.21.15.88:0/3392923887 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8a7414f800 msgr2=0x7f8a7414fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:41.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.418+0000 7f8a78a0e700 1 --2- 172.21.15.88:0/3392923887 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8a7414f800 0x7f8a7414fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:41.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.418+0000 7f8a78a0e700 1 -- 172.21.15.88:0/3392923887 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8a7414ce60 msgr2=0x7f8a7414d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:41.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.418+0000 7f8a78a0e700 1 --2- 172.21.15.88:0/3392923887 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8a7414ce60 0x7f8a7414d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:41.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.418+0000 7f8a78a0e700 1 -- 172.21.15.88:0/3392923887 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a7414f370 con 0x7f8a5c0038e0 2023-10-18T16:08:41.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.418+0000 7f8a7920f700 1 --2- 172.21.15.88:0/3392923887 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8a7414ce60 0x7f8a7414d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:08:41.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.418+0000 7f8a78a0e700 1 --2- 172.21.15.88:0/3392923887 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a5c0038e0 0x7f8a74161740 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f8a70006bc0 tx=0x7f8a70015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:08:41.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.418+0000 7f8a68ff9700 1 -- 172.21.15.88:0/3392923887 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a70006fd0 con 0x7f8a5c0038e0 2023-10-18T16:08:41.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.418+0000 7f8a68ff9700 1 -- 172.21.15.88:0/3392923887 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a70013670 con 0x7f8a5c0038e0 2023-10-18T16:08:41.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.418+0000 7f8a7ac72700 1 -- 172.21.15.88:0/3392923887 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a740b75d0 con 0x7f8a5c0038e0 2023-10-18T16:08:41.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.418+0000 7f8a68ff9700 1 -- 172.21.15.88:0/3392923887 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a70013a30 con 0x7f8a5c0038e0 2023-10-18T16:08:41.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.418+0000 7f8a7ac72700 1 -- 172.21.15.88:0/3392923887 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a740b7b90 con 0x7f8a5c0038e0 2023-10-18T16:08:41.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.419+0000 7f8a7ac72700 1 -- 172.21.15.88:0/3392923887 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a7406c4e0 con 0x7f8a5c0038e0 2023-10-18T16:08:41.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.419+0000 7f8a68ff9700 1 -- 172.21.15.88:0/3392923887 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f8a7001f430 con 0x7f8a5c0038e0 2023-10-18T16:08:41.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.420+0000 7f8a68ff9700 1 --2- 172.21.15.88:0/3392923887 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f8a6005f120 0x7f8a600615e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:41.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.420+0000 7f8a68ff9700 1 -- 172.21.15.88:0/3392923887 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6381+0+0 (secure 0 0 0) 0x7f8a7007ad70 con 0x7f8a5c0038e0 2023-10-18T16:08:41.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.420+0000 7f8a6bfff700 1 --2- 172.21.15.88:0/3392923887 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f8a6005f120 0x7f8a600615e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:41.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.420+0000 7f8a6bfff700 1 --2- 172.21.15.88:0/3392923887 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f8a6005f120 0x7f8a600615e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f8a640015a0 tx=0x7f8a64009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:08:41.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.424+0000 7f8a68ff9700 1 -- 172.21.15.88:0/3392923887 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f8a7004bee0 con 0x7f8a5c0038e0 2023-10-18T16:08:41.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.628+0000 7f8a7ac72700 1 -- 172.21.15.88:0/3392923887 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f8a7406a3a0 con 0x7f8a5c0038e0 2023-10-18T16:08:41.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.628+0000 7f8a68ff9700 1 -- 172.21.15.88:0/3392923887 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7f8a70013e60 con 0x7f8a5c0038e0 2023-10-18T16:08:41.633 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:08:41.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.630+0000 7f8a7ac72700 1 -- 172.21.15.88:0/3392923887 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f8a6005f120 msgr2=0x7f8a600615e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:41.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.630+0000 7f8a7ac72700 1 --2- 172.21.15.88:0/3392923887 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f8a6005f120 0x7f8a600615e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f8a640015a0 tx=0x7f8a64009040).stop 2023-10-18T16:08:41.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.630+0000 7f8a7ac72700 1 -- 172.21.15.88:0/3392923887 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a5c0038e0 msgr2=0x7f8a74161740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:41.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.630+0000 7f8a7ac72700 1 --2- 172.21.15.88:0/3392923887 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a5c0038e0 0x7f8a74161740 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f8a70006bc0 tx=0x7f8a70015000).stop 2023-10-18T16:08:41.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.630+0000 7f8a7ac72700 1 -- 172.21.15.88:0/3392923887 shutdown_connections 2023-10-18T16:08:41.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.630+0000 7f8a7ac72700 1 --2- 172.21.15.88:0/3392923887 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f8a6005f120 0x7f8a600615e0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:41.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.630+0000 7f8a7ac72700 1 --2- 172.21.15.88:0/3392923887 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8a7414ce60 0x7f8a7414d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:41.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.630+0000 7f8a7ac72700 1 --2- 172.21.15.88:0/3392923887 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a5c0038e0 0x7f8a74161740 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:41.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.630+0000 7f8a7ac72700 1 --2- 172.21.15.88:0/3392923887 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8a7414f800 0x7f8a7414fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:41.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.630+0000 7f8a7ac72700 1 -- 172.21.15.88:0/3392923887 >> 172.21.15.88:0/3392923887 conn(0x7f8a74063760 msgr2=0x7f8a740b6760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:08:41.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.630+0000 7f8a7ac72700 1 -- 172.21.15.88:0/3392923887 shutdown_connections 2023-10-18T16:08:41.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:41.630+0000 7f8a7ac72700 1 -- 172.21.15.88:0/3392923887 wait complete. 2023-10-18T16:08:41.645 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:08:43.885 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~54s 2023-10-18T16:08:43.886 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.f is failed for ~7s 2023-10-18T16:08:44.646 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:08:44.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.768+0000 7f8925cfb700 1 Processor -- start 2023-10-18T16:08:44.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.768+0000 7f8925cfb700 1 -- start start 2023-10-18T16:08:44.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.768+0000 7f8925cfb700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f892014d8c0 0x7f8920064440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:44.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.768+0000 7f8925cfb700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8920065360 0x7f8920064930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:44.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.768+0000 7f8925cfb700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8920064150 con 0x7f89200a9050 2023-10-18T16:08:44.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.768+0000 7f8925cfb700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8920153a40 con 0x7f8920065360 2023-10-18T16:08:44.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.768+0000 7f8925cfb700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8920063e70 con 0x7f892014d8c0 2023-10-18T16:08:44.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.769+0000 7f891ffff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8920065360 0x7f8920064930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:44.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.769+0000 7f891effd700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f89200a9050 0x7f8920153ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:59908/0 (socket says 172.21.15.88:59908) 2023-10-18T16:08:44.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.769+0000 7f891effd700 1 -- 172.21.15.88:0/857809958 learned_addr learned my addr 172.21.15.88:0/857809958 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:08:44.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.769+0000 7f891f7fe700 1 --2- 172.21.15.88:0/857809958 >> v2:172.21.15.151:3300/0 conn(0x7f892014d8c0 0x7f8920064440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:44.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.769+0000 7f891ffff700 1 -- 172.21.15.88:0/857809958 >> v2:172.21.15.151:3300/0 conn(0x7f892014d8c0 msgr2=0x7f8920064440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:08:44.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.769+0000 7f891ffff700 1 --2- 172.21.15.88:0/857809958 >> v2:172.21.15.151:3300/0 conn(0x7f892014d8c0 0x7f8920064440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:44.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.769+0000 7f891ffff700 1 -- 172.21.15.88:0/857809958 >> v1:172.21.15.88:6789/0 conn(0x7f89200a9050 legacy=0x7f8920153ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:44.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.769+0000 7f891ffff700 1 -- 172.21.15.88:0/857809958 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89200c72c0 con 0x7f8920065360 2023-10-18T16:08:44.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.769+0000 7f891ffff700 1 --2- 172.21.15.88:0/857809958 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8920065360 0x7f8920064930 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f8920062d50 tx=0x7f8918009780).ready entity=mon.0 client_cookie=10c92cac4c03823c server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:08:44.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.769+0000 7f891e7fc700 1 -- 172.21.15.88:0/857809958 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f891800e070 con 0x7f8920065360 2023-10-18T16:08:44.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.769+0000 7f891e7fc700 1 -- 172.21.15.88:0/857809958 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f891800bc30 con 0x7f8920065360 2023-10-18T16:08:44.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.769+0000 7f891e7fc700 1 -- 172.21.15.88:0/857809958 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8918012410 con 0x7f8920065360 2023-10-18T16:08:44.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.770+0000 7f8925cfb700 1 -- 172.21.15.88:0/857809958 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8920065360 msgr2=0x7f8920064930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:44.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.770+0000 7f8925cfb700 1 --2- 172.21.15.88:0/857809958 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8920065360 0x7f8920064930 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f8920062d50 tx=0x7f8918009780).stop 2023-10-18T16:08:44.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.770+0000 7f8925cfb700 1 -- 172.21.15.88:0/857809958 shutdown_connections 2023-10-18T16:08:44.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.770+0000 7f8925cfb700 1 --2- 172.21.15.88:0/857809958 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8920065360 0x7f8920064930 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:44.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.770+0000 7f8925cfb700 1 --2- 172.21.15.88:0/857809958 >> v2:172.21.15.151:3300/0 conn(0x7f892014d8c0 0x7f8920064440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:44.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.770+0000 7f8925cfb700 1 -- 172.21.15.88:0/857809958 >> 172.21.15.88:0/857809958 conn(0x7f892006a810 msgr2=0x7f89200c6a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:08:44.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.770+0000 7f8925cfb700 1 -- 172.21.15.88:0/857809958 shutdown_connections 2023-10-18T16:08:44.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.770+0000 7f8925cfb700 1 -- 172.21.15.88:0/857809958 wait complete. 2023-10-18T16:08:44.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.771+0000 7f8925cfb700 1 Processor -- start 2023-10-18T16:08:44.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.771+0000 7f8925cfb700 1 -- start start 2023-10-18T16:08:44.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.772+0000 7f8925cfb700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8920065360 0x7f89200aa9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:44.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.772+0000 7f8925cfb700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f89200a9050 0x7f8920149050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:44.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.772+0000 7f8925cfb700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f892014d8c0 0x7f892015fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:44.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.772+0000 7f8925cfb700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89200bfeb0 con 0x7f89200a9050 2023-10-18T16:08:44.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.772+0000 7f8925cfb700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8920160c20 con 0x7f8920065360 2023-10-18T16:08:44.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.772+0000 7f8925cfb700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89200c0a60 con 0x7f892014d8c0 2023-10-18T16:08:44.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.772+0000 7f891effd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f89200a9050 0x7f8920149050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:44.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.772+0000 7f891effd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f89200a9050 0x7f8920149050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40490/0 (socket says 172.21.15.88:40490) 2023-10-18T16:08:44.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.772+0000 7f891effd700 1 -- 172.21.15.88:0/2926200076 learned_addr learned my addr 172.21.15.88:0/2926200076 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:08:44.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.772+0000 7f891f7fe700 1 --2- 172.21.15.88:0/2926200076 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8920065360 0x7f89200aa9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:44.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.772+0000 7f891ffff700 1 --2- 172.21.15.88:0/2926200076 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f892014d8c0 0x7f892015fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:44.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.772+0000 7f891effd700 1 -- 172.21.15.88:0/2926200076 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f892014d8c0 msgr2=0x7f892015fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:44.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.772+0000 7f891effd700 1 --2- 172.21.15.88:0/2926200076 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f892014d8c0 0x7f892015fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:44.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.772+0000 7f891effd700 1 -- 172.21.15.88:0/2926200076 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8920065360 msgr2=0x7f89200aa9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:44.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.772+0000 7f891effd700 1 --2- 172.21.15.88:0/2926200076 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8920065360 0x7f89200aa9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:44.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.772+0000 7f891effd700 1 -- 172.21.15.88:0/2926200076 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8918009a20 con 0x7f89200a9050 2023-10-18T16:08:44.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.772+0000 7f891effd700 1 --2- 172.21.15.88:0/2926200076 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f89200a9050 0x7f8920149050 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f891000c490 tx=0x7f8910007df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:08:44.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.773+0000 7f89177fe700 1 -- 172.21.15.88:0/2926200076 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f891000ce00 con 0x7f89200a9050 2023-10-18T16:08:44.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.773+0000 7f89177fe700 1 -- 172.21.15.88:0/2926200076 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89100072e0 con 0x7f89200a9050 2023-10-18T16:08:44.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.773+0000 7f8925cfb700 1 -- 172.21.15.88:0/2926200076 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89200c0d00 con 0x7f89200a9050 2023-10-18T16:08:44.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.773+0000 7f89177fe700 1 -- 172.21.15.88:0/2926200076 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8910007660 con 0x7f89200a9050 2023-10-18T16:08:44.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.773+0000 7f8925cfb700 1 -- 172.21.15.88:0/2926200076 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89200bf860 con 0x7f89200a9050 2023-10-18T16:08:44.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.774+0000 7f8925cfb700 1 -- 172.21.15.88:0/2926200076 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89200c0060 con 0x7f89200a9050 2023-10-18T16:08:44.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.774+0000 7f89177fe700 1 -- 172.21.15.88:0/2926200076 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f89100059b0 con 0x7f89200a9050 2023-10-18T16:08:44.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.774+0000 7f89177fe700 1 --2- 172.21.15.88:0/2926200076 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f890005f150 0x7f8900061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:44.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.774+0000 7f89177fe700 1 -- 172.21.15.88:0/2926200076 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6381+0+0 (secure 0 0 0) 0x7f8910079910 con 0x7f89200a9050 2023-10-18T16:08:44.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.775+0000 7f891f7fe700 1 --2- 172.21.15.88:0/2926200076 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f890005f150 0x7f8900061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:44.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.775+0000 7f891f7fe700 1 --2- 172.21.15.88:0/2926200076 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f890005f150 0x7f8900061610 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f890800bb10 tx=0x7f8908008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:08:44.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.778+0000 7f89177fe700 1 -- 172.21.15.88:0/2926200076 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f89200c0060 con 0x7f89200a9050 2023-10-18T16:08:44.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.980+0000 7f8925cfb700 1 -- 172.21.15.88:0/2926200076 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f892006c4e0 con 0x7f89200a9050 2023-10-18T16:08:44.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.981+0000 7f89177fe700 1 -- 172.21.15.88:0/2926200076 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7f891004aae0 con 0x7f89200a9050 2023-10-18T16:08:44.985 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:08:44.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.983+0000 7f8925cfb700 1 -- 172.21.15.88:0/2926200076 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f890005f150 msgr2=0x7f8900061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:44.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.983+0000 7f8925cfb700 1 --2- 172.21.15.88:0/2926200076 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f890005f150 0x7f8900061610 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f890800bb10 tx=0x7f8908008040).stop 2023-10-18T16:08:44.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.983+0000 7f8925cfb700 1 -- 172.21.15.88:0/2926200076 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f89200a9050 msgr2=0x7f8920149050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:44.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.983+0000 7f8925cfb700 1 --2- 172.21.15.88:0/2926200076 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f89200a9050 0x7f8920149050 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f891000c490 tx=0x7f8910007df0).stop 2023-10-18T16:08:44.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.983+0000 7f8925cfb700 1 -- 172.21.15.88:0/2926200076 shutdown_connections 2023-10-18T16:08:44.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.983+0000 7f8925cfb700 1 --2- 172.21.15.88:0/2926200076 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f890005f150 0x7f8900061610 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:44.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.983+0000 7f8925cfb700 1 --2- 172.21.15.88:0/2926200076 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f892014d8c0 0x7f892015fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:44.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.983+0000 7f8925cfb700 1 --2- 172.21.15.88:0/2926200076 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8920065360 0x7f89200aa9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:44.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.983+0000 7f8925cfb700 1 --2- 172.21.15.88:0/2926200076 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f89200a9050 0x7f8920149050 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:44.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.983+0000 7f8925cfb700 1 -- 172.21.15.88:0/2926200076 >> 172.21.15.88:0/2926200076 conn(0x7f892006a810 msgr2=0x7f8920151810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:08:44.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.983+0000 7f8925cfb700 1 -- 172.21.15.88:0/2926200076 shutdown_connections 2023-10-18T16:08:44.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:44.983+0000 7f8925cfb700 1 -- 172.21.15.88:0/2926200076 wait complete. 2023-10-18T16:08:44.998 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:08:47.999 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:08:48.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.120+0000 7f0eb748a700 1 Processor -- start 2023-10-18T16:08:48.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.121+0000 7f0eb748a700 1 -- start start 2023-10-18T16:08:48.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.121+0000 7f0eb748a700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0eb014d8c0 0x7f0eb0064440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:48.131 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.121+0000 7f0eb748a700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f0eb0065360 0x7f0eb0064930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:48.131 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.121+0000 7f0eb748a700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0eb0064150 con 0x7f0eb00a9050 2023-10-18T16:08:48.131 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.121+0000 7f0eb748a700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0eb0153a40 con 0x7f0eb0065360 2023-10-18T16:08:48.131 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.121+0000 7f0eb748a700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb0063e70 con 0x7f0eb014d8c0 2023-10-18T16:08:48.131 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.122+0000 7f0eb5a27700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f0eb0065360 0x7f0eb0064930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:48.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.122+0000 7f0eb5226700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0eb014d8c0 0x7f0eb0064440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:48.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.122+0000 7f0eb5a27700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f0eb0065360 0x7f0eb0064930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.88:44194/0 (socket says 172.21.15.88:44194) 2023-10-18T16:08:48.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.122+0000 7f0eb5226700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0eb014d8c0 0x7f0eb0064440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.88:38936/0 (socket says 172.21.15.88:38936) 2023-10-18T16:08:48.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.122+0000 7f0eb5a27700 1 -- 172.21.15.88:0/3868199656 learned_addr learned my addr 172.21.15.88:0/3868199656 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:08:48.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.122+0000 7f0eb4a25700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f0eb00a9050 0x7f0eb0153ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.88:41684/0 (socket says 172.21.15.88:41684) 2023-10-18T16:08:48.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.122+0000 7f0eb5a27700 1 -- 172.21.15.88:0/3868199656 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0eb014d8c0 msgr2=0x7f0eb0064440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:08:48.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.122+0000 7f0eb5a27700 1 --2- 172.21.15.88:0/3868199656 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0eb014d8c0 0x7f0eb0064440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:48.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.122+0000 7f0eb5a27700 1 -- 172.21.15.88:0/3868199656 >> v1:172.21.15.151:6789/0 conn(0x7f0eb00a9050 legacy=0x7f0eb0153ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:48.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.122+0000 7f0eb5a27700 1 -- 172.21.15.88:0/3868199656 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eb00c72c0 con 0x7f0eb0065360 2023-10-18T16:08:48.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.122+0000 7f0eb5226700 1 --2- 172.21.15.88:0/3868199656 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0eb014d8c0 0x7f0eb0064440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2023-10-18T16:08:48.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.122+0000 7f0eb5a27700 1 --2- 172.21.15.88:0/3868199656 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0eb0065360 0x7f0eb0064930 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f0ea8006fd0 tx=0x7f0ea80090f0).ready entity=mon.1 client_cookie=a4d3d9038d4c3bc6 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:08:48.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.123+0000 7f0ea7fff700 1 -- 172.21.15.88:0/3868199656 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea800f510 con 0x7f0eb0065360 2023-10-18T16:08:48.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.123+0000 7f0ea7fff700 1 -- 172.21.15.88:0/3868199656 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ea800f670 con 0x7f0eb0065360 2023-10-18T16:08:48.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.123+0000 7f0ea7fff700 1 -- 172.21.15.88:0/3868199656 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea800fa10 con 0x7f0eb0065360 2023-10-18T16:08:48.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.123+0000 7f0eb748a700 1 -- 172.21.15.88:0/3868199656 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0eb0065360 msgr2=0x7f0eb0064930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:48.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.123+0000 7f0eb748a700 1 --2- 172.21.15.88:0/3868199656 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0eb0065360 0x7f0eb0064930 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f0ea8006fd0 tx=0x7f0ea80090f0).stop 2023-10-18T16:08:48.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.123+0000 7f0eb748a700 1 -- 172.21.15.88:0/3868199656 shutdown_connections 2023-10-18T16:08:48.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.123+0000 7f0eb748a700 1 --2- 172.21.15.88:0/3868199656 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0eb0065360 0x7f0eb0064930 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:48.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.123+0000 7f0eb748a700 1 --2- 172.21.15.88:0/3868199656 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0eb014d8c0 0x7f0eb0064440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:48.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.123+0000 7f0eb748a700 1 -- 172.21.15.88:0/3868199656 >> 172.21.15.88:0/3868199656 conn(0x7f0eb006a810 msgr2=0x7f0eb00c6a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:08:48.136 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.123+0000 7f0eb748a700 1 -- 172.21.15.88:0/3868199656 shutdown_connections 2023-10-18T16:08:48.136 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.123+0000 7f0eb748a700 1 -- 172.21.15.88:0/3868199656 wait complete. 2023-10-18T16:08:48.136 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.124+0000 7f0eb748a700 1 Processor -- start 2023-10-18T16:08:48.136 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.124+0000 7f0eb748a700 1 -- start start 2023-10-18T16:08:48.136 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.125+0000 7f0eb748a700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0eb0065360 0x7f0eb014cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:48.137 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.125+0000 7f0eb748a700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0eb00a9050 0x7f0eb00bcba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:48.137 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.125+0000 7f0eb748a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0eb014d8c0 0x7f0eb00a9a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:48.137 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.125+0000 7f0eb748a700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb014f9e0 con 0x7f0eb00a9050 2023-10-18T16:08:48.137 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.125+0000 7f0eb748a700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb00aad80 con 0x7f0eb014d8c0 2023-10-18T16:08:48.137 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.125+0000 7f0eb748a700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb00acac0 con 0x7f0eb0065360 2023-10-18T16:08:48.138 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.125+0000 7f0eb4a25700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0eb00a9050 0x7f0eb00bcba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:48.138 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.125+0000 7f0eb4a25700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0eb00a9050 0x7f0eb00bcba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:39630/0 (socket says 172.21.15.88:39630) 2023-10-18T16:08:48.138 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.125+0000 7f0eb4a25700 1 -- 172.21.15.88:0/257227168 learned_addr learned my addr 172.21.15.88:0/257227168 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:08:48.138 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.126+0000 7f0eb5a27700 1 --2- 172.21.15.88:0/257227168 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0eb014d8c0 0x7f0eb00a9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:48.138 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.126+0000 7f0eb5226700 1 --2- 172.21.15.88:0/257227168 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0eb0065360 0x7f0eb014cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:48.139 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.126+0000 7f0eb4a25700 1 -- 172.21.15.88:0/257227168 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0eb0065360 msgr2=0x7f0eb014cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:48.139 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.126+0000 7f0eb4a25700 1 --2- 172.21.15.88:0/257227168 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0eb0065360 0x7f0eb014cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:48.139 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.126+0000 7f0eb4a25700 1 -- 172.21.15.88:0/257227168 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0eb014d8c0 msgr2=0x7f0eb00a9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:48.139 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.126+0000 7f0eb4a25700 1 --2- 172.21.15.88:0/257227168 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0eb014d8c0 0x7f0eb00a9a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:48.139 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.126+0000 7f0eb4a25700 1 -- 172.21.15.88:0/257227168 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eac009a20 con 0x7f0eb00a9050 2023-10-18T16:08:48.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.126+0000 7f0eb5a27700 1 --2- 172.21.15.88:0/257227168 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0eb014d8c0 0x7f0eb00a9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-10-18T16:08:48.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.126+0000 7f0eb4a25700 1 --2- 172.21.15.88:0/257227168 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0eb00a9050 0x7f0eb00bcba0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f0ea000c600 tx=0x7f0ea0007d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:08:48.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.126+0000 7f0ea57fa700 1 -- 172.21.15.88:0/257227168 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea0007320 con 0x7f0eb00a9050 2023-10-18T16:08:48.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.126+0000 7f0ea57fa700 1 -- 172.21.15.88:0/257227168 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ea0007480 con 0x7f0eb00a9050 2023-10-18T16:08:48.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.126+0000 7f0eb748a700 1 -- 172.21.15.88:0/257227168 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ea8009370 con 0x7f0eb00a9050 2023-10-18T16:08:48.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.126+0000 7f0ea57fa700 1 -- 172.21.15.88:0/257227168 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea00058f0 con 0x7f0eb00a9050 2023-10-18T16:08:48.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.126+0000 7f0eb748a700 1 -- 172.21.15.88:0/257227168 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0eb0160af0 con 0x7f0eb00a9050 2023-10-18T16:08:48.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.127+0000 7f0eb748a700 1 -- 172.21.15.88:0/257227168 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0eb01615d0 con 0x7f0eb00a9050 2023-10-18T16:08:48.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.127+0000 7f0ea57fa700 1 -- 172.21.15.88:0/257227168 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f0ea0005ca0 con 0x7f0eb00a9050 2023-10-18T16:08:48.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.128+0000 7f0ea57fa700 1 --2- 172.21.15.88:0/257227168 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f0e9405f150 0x7f0e94061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:48.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.128+0000 7f0ea57fa700 1 -- 172.21.15.88:0/257227168 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6381+0+0 (secure 0 0 0) 0x7f0ea007a1d0 con 0x7f0eb00a9050 2023-10-18T16:08:48.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.128+0000 7f0eb5226700 1 --2- 172.21.15.88:0/257227168 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f0e9405f150 0x7f0e94061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:48.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.128+0000 7f0eb5226700 1 --2- 172.21.15.88:0/257227168 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f0e9405f150 0x7f0e94061610 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f0eac0099f0 tx=0x7f0eac00b040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:08:48.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.132+0000 7f0ea57fa700 1 -- 172.21.15.88:0/257227168 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f0eb01615d0 con 0x7f0eb00a9050 2023-10-18T16:08:48.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.330+0000 7f0eb748a700 1 -- 172.21.15.88:0/257227168 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f0eb006c4e0 con 0x7f0eb00a9050 2023-10-18T16:08:48.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.330+0000 7f0ea57fa700 1 -- 172.21.15.88:0/257227168 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7f0ea004b3a0 con 0x7f0eb00a9050 2023-10-18T16:08:48.334 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:08:48.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.332+0000 7f0eb748a700 1 -- 172.21.15.88:0/257227168 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f0e9405f150 msgr2=0x7f0e94061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:48.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.332+0000 7f0eb748a700 1 --2- 172.21.15.88:0/257227168 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f0e9405f150 0x7f0e94061610 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f0eac0099f0 tx=0x7f0eac00b040).stop 2023-10-18T16:08:48.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.332+0000 7f0eb748a700 1 -- 172.21.15.88:0/257227168 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0eb00a9050 msgr2=0x7f0eb00bcba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:48.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.332+0000 7f0eb748a700 1 --2- 172.21.15.88:0/257227168 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0eb00a9050 0x7f0eb00bcba0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f0ea000c600 tx=0x7f0ea0007d80).stop 2023-10-18T16:08:48.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.332+0000 7f0eb748a700 1 -- 172.21.15.88:0/257227168 shutdown_connections 2023-10-18T16:08:48.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.332+0000 7f0eb748a700 1 --2- 172.21.15.88:0/257227168 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f0e9405f150 0x7f0e94061610 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:48.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.332+0000 7f0eb748a700 1 --2- 172.21.15.88:0/257227168 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0eb014d8c0 0x7f0eb00a9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:48.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.332+0000 7f0eb748a700 1 --2- 172.21.15.88:0/257227168 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0eb0065360 0x7f0eb014cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:48.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.332+0000 7f0eb748a700 1 --2- 172.21.15.88:0/257227168 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0eb00a9050 0x7f0eb00bcba0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:48.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.332+0000 7f0eb748a700 1 -- 172.21.15.88:0/257227168 >> 172.21.15.88:0/257227168 conn(0x7f0eb006a810 msgr2=0x7f0eb0153ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:08:48.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.332+0000 7f0eb748a700 1 -- 172.21.15.88:0/257227168 shutdown_connections 2023-10-18T16:08:48.338 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:48.332+0000 7f0eb748a700 1 -- 172.21.15.88:0/257227168 wait complete. 2023-10-18T16:08:48.347 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:08:48.429 INFO:tasks.ceph.mds.e.smithi088.stderr:terminate called after throwing an instance of 'boost::wrapexcept' 2023-10-18T16:08:48.429 INFO:tasks.ceph.mds.e.smithi088.stderr: what(): boost::bad_get: failed value get using boost::get 2023-10-18T16:08:48.430 INFO:tasks.ceph.mds.e.smithi088.stderr:*** Caught signal (Aborted) ** 2023-10-18T16:08:48.430 INFO:tasks.ceph.mds.e.smithi088.stderr: in thread 7ff460c00700 thread_name:ms_dispatch 2023-10-18T16:08:48.430 INFO:tasks.ceph.mds.e.smithi088.stderr: ceph version 16.2.14-100-gb4ddd52c (b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5) pacific (stable) 2023-10-18T16:08:48.431 INFO:tasks.ceph.mds.e.smithi088.stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7ff468403b20] 2023-10-18T16:08:48.431 INFO:tasks.ceph.mds.e.smithi088.stderr: 2: gsignal() 2023-10-18T16:08:48.431 INFO:tasks.ceph.mds.e.smithi088.stderr: 3: abort() 2023-10-18T16:08:48.431 INFO:tasks.ceph.mds.e.smithi088.stderr: 4: /lib64/libstdc++.so.6(+0x9009b) [0x7ff46780409b] 2023-10-18T16:08:48.431 INFO:tasks.ceph.mds.e.smithi088.stderr: 5: /lib64/libstdc++.so.6(+0x9653c) [0x7ff46780a53c] 2023-10-18T16:08:48.432 INFO:tasks.ceph.mds.e.smithi088.stderr: 6: /lib64/libstdc++.so.6(+0x96597) [0x7ff46780a597] 2023-10-18T16:08:48.432 INFO:tasks.ceph.mds.e.smithi088.stderr: 7: /lib64/libstdc++.so.6(+0x967f8) [0x7ff46780a7f8] 2023-10-18T16:08:48.432 INFO:tasks.ceph.mds.e.smithi088.stderr: 8: ceph-mds(+0x212b43) [0x5631a209bb43] 2023-10-18T16:08:48.432 INFO:tasks.ceph.mds.e.smithi088.stderr: 9: (Option::size_t const md_config_t::get_val(ConfigValues const&, std::basic_string_view >) const+0x5c) [0x5631a211089c] 2023-10-18T16:08:48.432 INFO:tasks.ceph.mds.e.smithi088.stderr: 10: (SessionMap::SessionMap(MDSRank*)+0x2c2) [0x5631a23586a2] 2023-10-18T16:08:48.433 INFO:tasks.ceph.mds.e.smithi088.stderr: 11: (MDSRank::MDSRank(int, std::__cxx11::basic_string, std::allocator >, ceph::fair_mutex&, std::shared_ptr&, CommonSafeTimer&, Beacon&, std::unique_ptr >&, Messenger*, MonClient*, MgrClient*, Context*, Context*, boost::asio::io_context&)+0x214) [0x5631a2075724] 2023-10-18T16:08:48.433 INFO:tasks.ceph.mds.e.smithi088.stderr: 12: (MDSRankDispatcher::MDSRankDispatcher(int, std::__cxx11::basic_string, std::allocator >, ceph::fair_mutex&, std::shared_ptr&, CommonSafeTimer&, Beacon&, std::unique_ptr >&, Messenger*, MonClient*, MgrClient*, Context*, Context*, boost::asio::io_context&)+0x101) [0x5631a2077181] 2023-10-18T16:08:48.433 INFO:tasks.ceph.mds.e.smithi088.stderr: 13: (MDSDaemon::handle_mds_map(boost::intrusive_ptr const&)+0x1608) [0x5631a2047348] 2023-10-18T16:08:48.433 INFO:tasks.ceph.mds.e.smithi088.stderr: 14: (MDSDaemon::handle_core_message(boost::intrusive_ptr const&)+0xcd) [0x5631a204a44d] 2023-10-18T16:08:48.433 INFO:tasks.ceph.mds.e.smithi088.stderr: 15: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0xc3) [0x5631a204afa3] 2023-10-18T16:08:48.433 INFO:tasks.ceph.mds.e.smithi088.stderr: 16: (DispatchQueue::entry()+0x126a) [0x7ff46965d4ea] 2023-10-18T16:08:48.434 INFO:tasks.ceph.mds.e.smithi088.stderr: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff469711d51] 2023-10-18T16:08:48.434 INFO:tasks.ceph.mds.e.smithi088.stderr: 18: /lib64/libpthread.so.0(+0x814a) [0x7ff4683f914a] 2023-10-18T16:08:48.434 INFO:tasks.ceph.mds.e.smithi088.stderr: 19: clone() 2023-10-18T16:08:48.434 INFO:tasks.ceph.mds.e.smithi088.stderr:2023-10-18T16:08:48.427+0000 7ff460c00700 -1 *** Caught signal (Aborted) ** 2023-10-18T16:08:48.435 INFO:tasks.ceph.mds.e.smithi088.stderr: in thread 7ff460c00700 thread_name:ms_dispatch 2023-10-18T16:08:48.435 INFO:tasks.ceph.mds.e.smithi088.stderr: 2023-10-18T16:08:48.435 INFO:tasks.ceph.mds.e.smithi088.stderr: ceph version 16.2.14-100-gb4ddd52c (b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5) pacific (stable) 2023-10-18T16:08:48.435 INFO:tasks.ceph.mds.e.smithi088.stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7ff468403b20] 2023-10-18T16:08:48.435 INFO:tasks.ceph.mds.e.smithi088.stderr: 2: gsignal() 2023-10-18T16:08:48.435 INFO:tasks.ceph.mds.e.smithi088.stderr: 3: abort() 2023-10-18T16:08:48.436 INFO:tasks.ceph.mds.e.smithi088.stderr: 4: /lib64/libstdc++.so.6(+0x9009b) [0x7ff46780409b] 2023-10-18T16:08:48.436 INFO:tasks.ceph.mds.e.smithi088.stderr: 5: /lib64/libstdc++.so.6(+0x9653c) [0x7ff46780a53c] 2023-10-18T16:08:48.436 INFO:tasks.ceph.mds.e.smithi088.stderr: 6: /lib64/libstdc++.so.6(+0x96597) [0x7ff46780a597] 2023-10-18T16:08:48.436 INFO:tasks.ceph.mds.e.smithi088.stderr: 7: /lib64/libstdc++.so.6(+0x967f8) [0x7ff46780a7f8] 2023-10-18T16:08:48.436 INFO:tasks.ceph.mds.e.smithi088.stderr: 8: ceph-mds(+0x212b43) [0x5631a209bb43] 2023-10-18T16:08:48.437 INFO:tasks.ceph.mds.e.smithi088.stderr: 9: (Option::size_t const md_config_t::get_val(ConfigValues const&, std::basic_string_view >) const+0x5c) [0x5631a211089c] 2023-10-18T16:08:48.437 INFO:tasks.ceph.mds.e.smithi088.stderr: 10: (SessionMap::SessionMap(MDSRank*)+0x2c2) [0x5631a23586a2] 2023-10-18T16:08:48.437 INFO:tasks.ceph.mds.e.smithi088.stderr: 11: (MDSRank::MDSRank(int, std::__cxx11::basic_string, std::allocator >, ceph::fair_mutex&, std::shared_ptr&, CommonSafeTimer&, Beacon&, std::unique_ptr >&, Messenger*, MonClient*, MgrClient*, Context*, Context*, boost::asio::io_context&)+0x214) [0x5631a2075724] 2023-10-18T16:08:48.437 INFO:tasks.ceph.mds.e.smithi088.stderr: 12: (MDSRankDispatcher::MDSRankDispatcher(int, std::__cxx11::basic_string, std::allocator >, ceph::fair_mutex&, std::shared_ptr&, CommonSafeTimer&, Beacon&, std::unique_ptr >&, Messenger*, MonClient*, MgrClient*, Context*, Context*, boost::asio::io_context&)+0x101) [0x5631a2077181] 2023-10-18T16:08:48.438 INFO:tasks.ceph.mds.e.smithi088.stderr: 13: (MDSDaemon::handle_mds_map(boost::intrusive_ptr const&)+0x1608) [0x5631a2047348] 2023-10-18T16:08:48.438 INFO:tasks.ceph.mds.e.smithi088.stderr: 14: (MDSDaemon::handle_core_message(boost::intrusive_ptr const&)+0xcd) [0x5631a204a44d] 2023-10-18T16:08:48.438 INFO:tasks.ceph.mds.e.smithi088.stderr: 15: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0xc3) [0x5631a204afa3] 2023-10-18T16:08:48.438 INFO:tasks.ceph.mds.e.smithi088.stderr: 16: (DispatchQueue::entry()+0x126a) [0x7ff46965d4ea] 2023-10-18T16:08:48.438 INFO:tasks.ceph.mds.e.smithi088.stderr: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff469711d51] 2023-10-18T16:08:48.439 INFO:tasks.ceph.mds.e.smithi088.stderr: 18: /lib64/libpthread.so.0(+0x814a) [0x7ff4683f914a] 2023-10-18T16:08:48.439 INFO:tasks.ceph.mds.e.smithi088.stderr: 19: clone() 2023-10-18T16:08:48.439 INFO:tasks.ceph.mds.e.smithi088.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-10-18T16:08:48.439 INFO:tasks.ceph.mds.e.smithi088.stderr: 2023-10-18T16:08:48.440 INFO:tasks.ceph.mds.e.smithi088.stderr: 0> 2023-10-18T16:08:48.427+0000 7ff460c00700 -1 *** Caught signal (Aborted) ** 2023-10-18T16:08:48.440 INFO:tasks.ceph.mds.e.smithi088.stderr: in thread 7ff460c00700 thread_name:ms_dispatch 2023-10-18T16:08:48.440 INFO:tasks.ceph.mds.e.smithi088.stderr: 2023-10-18T16:08:48.440 INFO:tasks.ceph.mds.e.smithi088.stderr: ceph version 16.2.14-100-gb4ddd52c (b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5) pacific (stable) 2023-10-18T16:08:48.440 INFO:tasks.ceph.mds.e.smithi088.stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7ff468403b20] 2023-10-18T16:08:48.441 INFO:tasks.ceph.mds.e.smithi088.stderr: 2: gsignal() 2023-10-18T16:08:48.441 INFO:tasks.ceph.mds.e.smithi088.stderr: 3: abort() 2023-10-18T16:08:48.441 INFO:tasks.ceph.mds.e.smithi088.stderr: 4: /lib64/libstdc++.so.6(+0x9009b) [0x7ff46780409b] 2023-10-18T16:08:48.441 INFO:tasks.ceph.mds.e.smithi088.stderr: 5: /lib64/libstdc++.so.6(+0x9653c) [0x7ff46780a53c] 2023-10-18T16:08:48.441 INFO:tasks.ceph.mds.e.smithi088.stderr: 6: /lib64/libstdc++.so.6(+0x96597) [0x7ff46780a597] 2023-10-18T16:08:48.442 INFO:tasks.ceph.mds.e.smithi088.stderr: 7: /lib64/libstdc++.so.6(+0x967f8) [0x7ff46780a7f8] 2023-10-18T16:08:48.442 INFO:tasks.ceph.mds.e.smithi088.stderr: 8: ceph-mds(+0x212b43) [0x5631a209bb43] 2023-10-18T16:08:48.442 INFO:tasks.ceph.mds.e.smithi088.stderr: 9: (Option::size_t const md_config_t::get_val(ConfigValues const&, std::basic_string_view >) const+0x5c) [0x5631a211089c] 2023-10-18T16:08:48.442 INFO:tasks.ceph.mds.e.smithi088.stderr: 10: (SessionMap::SessionMap(MDSRank*)+0x2c2) [0x5631a23586a2] 2023-10-18T16:08:48.443 INFO:tasks.ceph.mds.e.smithi088.stderr: 11: (MDSRank::MDSRank(int, std::__cxx11::basic_string, std::allocator >, ceph::fair_mutex&, std::shared_ptr&, CommonSafeTimer&, Beacon&, std::unique_ptr >&, Messenger*, MonClient*, MgrClient*, Context*, Context*, boost::asio::io_context&)+0x214) [0x5631a2075724] 2023-10-18T16:08:48.443 INFO:tasks.ceph.mds.e.smithi088.stderr: 12: (MDSRankDispatcher::MDSRankDispatcher(int, std::__cxx11::basic_string, std::allocator >, ceph::fair_mutex&, std::shared_ptr&, CommonSafeTimer&, Beacon&, std::unique_ptr >&, Messenger*, MonClient*, MgrClient*, Context*, Context*, boost::asio::io_context&)+0x101) [0x5631a2077181] 2023-10-18T16:08:48.443 INFO:tasks.ceph.mds.e.smithi088.stderr: 13: (MDSDaemon::handle_mds_map(boost::intrusive_ptr const&)+0x1608) [0x5631a2047348] 2023-10-18T16:08:48.443 INFO:tasks.ceph.mds.e.smithi088.stderr: 14: (MDSDaemon::handle_core_message(boost::intrusive_ptr const&)+0xcd) [0x5631a204a44d] 2023-10-18T16:08:48.443 INFO:tasks.ceph.mds.e.smithi088.stderr: 15: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0xc3) [0x5631a204afa3] 2023-10-18T16:08:48.443 INFO:tasks.ceph.mds.e.smithi088.stderr: 16: (DispatchQueue::entry()+0x126a) [0x7ff46965d4ea] 2023-10-18T16:08:48.444 INFO:tasks.ceph.mds.e.smithi088.stderr: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff469711d51] 2023-10-18T16:08:48.444 INFO:tasks.ceph.mds.e.smithi088.stderr: 18: /lib64/libpthread.so.0(+0x814a) [0x7ff4683f914a] 2023-10-18T16:08:48.444 INFO:tasks.ceph.mds.e.smithi088.stderr: 19: clone() 2023-10-18T16:08:48.444 INFO:tasks.ceph.mds.e.smithi088.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-10-18T16:08:48.444 INFO:tasks.ceph.mds.e.smithi088.stderr: 2023-10-18T16:08:48.445 INFO:tasks.ceph.mds.e.smithi088.stderr: 0> 2023-10-18T16:08:48.427+0000 7ff460c00700 -1 *** Caught signal (Aborted) ** 2023-10-18T16:08:48.445 INFO:tasks.ceph.mds.e.smithi088.stderr: in thread 7ff460c00700 thread_name:ms_dispatch 2023-10-18T16:08:48.445 INFO:tasks.ceph.mds.e.smithi088.stderr: 2023-10-18T16:08:48.445 INFO:tasks.ceph.mds.e.smithi088.stderr: ceph version 16.2.14-100-gb4ddd52c (b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5) pacific (stable) 2023-10-18T16:08:48.446 INFO:tasks.ceph.mds.e.smithi088.stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7ff468403b20] 2023-10-18T16:08:48.446 INFO:tasks.ceph.mds.e.smithi088.stderr: 2: gsignal() 2023-10-18T16:08:48.446 INFO:tasks.ceph.mds.e.smithi088.stderr: 3: abort() 2023-10-18T16:08:48.446 INFO:tasks.ceph.mds.e.smithi088.stderr: 4: /lib64/libstdc++.so.6(+0x9009b) [0x7ff46780409b] 2023-10-18T16:08:48.446 INFO:tasks.ceph.mds.e.smithi088.stderr: 5: /lib64/libstdc++.so.6(+0x9653c) [0x7ff46780a53c] 2023-10-18T16:08:48.447 INFO:tasks.ceph.mds.e.smithi088.stderr: 6: /lib64/libstdc++.so.6(+0x96597) [0x7ff46780a597] 2023-10-18T16:08:48.447 INFO:tasks.ceph.mds.e.smithi088.stderr: 7: /lib64/libstdc++.so.6(+0x967f8) [0x7ff46780a7f8] 2023-10-18T16:08:48.447 INFO:tasks.ceph.mds.e.smithi088.stderr: 8: ceph-mds(+0x212b43) [0x5631a209bb43] 2023-10-18T16:08:48.447 INFO:tasks.ceph.mds.e.smithi088.stderr: 9: (Option::size_t const md_config_t::get_val(ConfigValues const&, std::basic_string_view >) const+0x5c) [0x5631a211089c] 2023-10-18T16:08:48.447 INFO:tasks.ceph.mds.e.smithi088.stderr: 10: (SessionMap::SessionMap(MDSRank*)+0x2c2) [0x5631a23586a2] 2023-10-18T16:08:48.447 INFO:tasks.ceph.mds.e.smithi088.stderr: 11: (MDSRank::MDSRank(int, std::__cxx11::basic_string, std::allocator >, ceph::fair_mutex&, std::shared_ptr&, CommonSafeTimer&, Beacon&, std::unique_ptr >&, Messenger*, MonClient*, MgrClient*, Context*, Context*, boost::asio::io_context&)+0x214) [0x5631a2075724] 2023-10-18T16:08:48.448 INFO:tasks.ceph.mds.e.smithi088.stderr: 12: (MDSRankDispatcher::MDSRankDispatcher(int, std::__cxx11::basic_string, std::allocator >, ceph::fair_mutex&, std::shared_ptr&, CommonSafeTimer&, Beacon&, std::unique_ptr >&, Messenger*, MonClient*, MgrClient*, Context*, Context*, boost::asio::io_context&)+0x101) [0x5631a2077181] 2023-10-18T16:08:48.448 INFO:tasks.ceph.mds.e.smithi088.stderr: 13: (MDSDaemon::handle_mds_map(boost::intrusive_ptr const&)+0x1608) [0x5631a2047348] 2023-10-18T16:08:48.448 INFO:tasks.ceph.mds.e.smithi088.stderr: 14: (MDSDaemon::handle_core_message(boost::intrusive_ptr const&)+0xcd) [0x5631a204a44d] 2023-10-18T16:08:48.448 INFO:tasks.ceph.mds.e.smithi088.stderr: 15: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0xc3) [0x5631a204afa3] 2023-10-18T16:08:48.448 INFO:tasks.ceph.mds.e.smithi088.stderr: 16: (DispatchQueue::entry()+0x126a) [0x7ff46965d4ea] 2023-10-18T16:08:48.449 INFO:tasks.ceph.mds.e.smithi088.stderr: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff469711d51] 2023-10-18T16:08:48.449 INFO:tasks.ceph.mds.e.smithi088.stderr: 18: /lib64/libpthread.so.0(+0x814a) [0x7ff4683f914a] 2023-10-18T16:08:48.449 INFO:tasks.ceph.mds.e.smithi088.stderr: 19: clone() 2023-10-18T16:08:48.449 INFO:tasks.ceph.mds.e.smithi088.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-10-18T16:08:48.449 INFO:tasks.ceph.mds.e.smithi088.stderr: 2023-10-18T16:08:48.720 INFO:tasks.ceph.mds.e.smithi088.stderr:daemon-helper: command crashed with signal 6 2023-10-18T16:08:50.497 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~0s 2023-10-18T16:08:50.497 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~61s 2023-10-18T16:08:50.497 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.f is failed for ~13s 2023-10-18T16:08:51.348 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:08:51.473 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.469+0000 7fc1d34e2700 1 Processor -- start 2023-10-18T16:08:51.473 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.469+0000 7fc1d34e2700 1 -- start start 2023-10-18T16:08:51.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.470+0000 7fc1d34e2700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fc1cc153010 0x7fc1cc1533f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:51.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.470+0000 7fc1d34e2700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc1cc1525f0 0x7fc1cc0c36c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:51.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.470+0000 7fc1d34e2700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1cc0a9670 con 0x7fc1cc064d70 2023-10-18T16:08:51.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.470+0000 7fc1d34e2700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1cc14df50 con 0x7fc1cc153010 2023-10-18T16:08:51.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.470+0000 7fc1d34e2700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1cc151a50 con 0x7fc1cc1525f0 2023-10-18T16:08:51.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.470+0000 7fc1d1a7f700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc1cc1525f0 0x7fc1cc0c36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:51.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.470+0000 7fc1d127e700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fc1cc153010 0x7fc1cc1533f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:51.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.471+0000 7fc1d0a7d700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fc1cc064d70 0x7fc1cc151ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.88:41694/0 (socket says 172.21.15.88:41694) 2023-10-18T16:08:51.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.471+0000 7fc1d1a7f700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc1cc1525f0 0x7fc1cc0c36c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.88:38950/0 (socket says 172.21.15.88:38950) 2023-10-18T16:08:51.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.471+0000 7fc1d0a7d700 1 -- 172.21.15.88:0/3509012956 learned_addr learned my addr 172.21.15.88:0/3509012956 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:08:51.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.471+0000 7fc1d127e700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fc1cc153010 0x7fc1cc1533f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.88:44218/0 (socket says 172.21.15.88:44218) 2023-10-18T16:08:51.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.471+0000 7fc1d1a7f700 1 -- 172.21.15.88:0/3509012956 >> v2:172.21.15.151:3300/0 conn(0x7fc1cc153010 msgr2=0x7fc1cc1533f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:08:51.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.471+0000 7fc1d1a7f700 1 --2- 172.21.15.88:0/3509012956 >> v2:172.21.15.151:3300/0 conn(0x7fc1cc153010 0x7fc1cc1533f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:51.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.471+0000 7fc1d1a7f700 1 -- 172.21.15.88:0/3509012956 >> v1:172.21.15.151:6789/0 conn(0x7fc1cc064d70 legacy=0x7fc1cc151ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:51.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.471+0000 7fc1c3fff700 1 -- 172.21.15.88:0/3509012956 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2475945819 0 0) 0x7fc1cc0a9670 con 0x7fc1cc064d70 2023-10-18T16:08:51.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.471+0000 7fc1d1a7f700 1 -- 172.21.15.88:0/3509012956 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1cc0a8af0 con 0x7fc1cc1525f0 2023-10-18T16:08:51.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.471+0000 7fc1d127e700 1 --2- 172.21.15.88:0/3509012956 >> v2:172.21.15.151:3300/0 conn(0x7fc1cc153010 0x7fc1cc1533f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2023-10-18T16:08:51.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.471+0000 7fc1d1a7f700 1 --2- 172.21.15.88:0/3509012956 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc1cc1525f0 0x7fc1cc0c36c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fc1c4009760 tx=0x7fc1c4009060).ready entity=mon.2 client_cookie=e97542367ccac8dc server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:08:51.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.471+0000 7fc1c3fff700 1 -- 172.21.15.88:0/3509012956 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c40090d0 con 0x7fc1cc1525f0 2023-10-18T16:08:51.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.471+0000 7fc1c3fff700 1 -- 172.21.15.88:0/3509012956 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1c400f510 con 0x7fc1cc1525f0 2023-10-18T16:08:51.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.471+0000 7fc1c3fff700 1 -- 172.21.15.88:0/3509012956 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c400f890 con 0x7fc1cc1525f0 2023-10-18T16:08:51.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.472+0000 7fc1d34e2700 1 -- 172.21.15.88:0/3509012956 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc1cc1525f0 msgr2=0x7fc1cc0c36c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:51.481 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.472+0000 7fc1d34e2700 1 --2- 172.21.15.88:0/3509012956 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc1cc1525f0 0x7fc1cc0c36c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fc1c4009760 tx=0x7fc1c4009060).stop 2023-10-18T16:08:51.481 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.472+0000 7fc1d34e2700 1 -- 172.21.15.88:0/3509012956 shutdown_connections 2023-10-18T16:08:51.481 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.472+0000 7fc1d34e2700 1 --2- 172.21.15.88:0/3509012956 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc1cc1525f0 0x7fc1cc0c36c0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:51.481 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.472+0000 7fc1d34e2700 1 --2- 172.21.15.88:0/3509012956 >> v2:172.21.15.151:3300/0 conn(0x7fc1cc153010 0x7fc1cc1533f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:51.481 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.472+0000 7fc1d34e2700 1 -- 172.21.15.88:0/3509012956 >> 172.21.15.88:0/3509012956 conn(0x7fc1cc069d80 msgr2=0x7fc1cc06a190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:08:51.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.472+0000 7fc1d34e2700 1 -- 172.21.15.88:0/3509012956 shutdown_connections 2023-10-18T16:08:51.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.472+0000 7fc1d34e2700 1 -- 172.21.15.88:0/3509012956 wait complete. 2023-10-18T16:08:51.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.473+0000 7fc1d34e2700 1 Processor -- start 2023-10-18T16:08:51.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.473+0000 7fc1d34e2700 1 -- start start 2023-10-18T16:08:51.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.474+0000 7fc1d34e2700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc1cc064d70 0x7fc1cc1632b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:51.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.474+0000 7fc1d34e2700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc1cc1525f0 0x7fc1cc1637a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:51.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.474+0000 7fc1d34e2700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1cc153010 0x7fc1cc0ab110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:51.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.474+0000 7fc1d34e2700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1cc146af0 con 0x7fc1cc153010 2023-10-18T16:08:51.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.474+0000 7fc1d34e2700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1cc163ef0 con 0x7fc1cc064d70 2023-10-18T16:08:51.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.474+0000 7fc1d34e2700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1cc14f3e0 con 0x7fc1cc1525f0 2023-10-18T16:08:51.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.474+0000 7fc1d1a7f700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1cc153010 0x7fc1cc0ab110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:51.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.474+0000 7fc1d1a7f700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1cc153010 0x7fc1cc0ab110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:39632/0 (socket says 172.21.15.88:39632) 2023-10-18T16:08:51.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.474+0000 7fc1d1a7f700 1 -- 172.21.15.88:0/1393443585 learned_addr learned my addr 172.21.15.88:0/1393443585 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:08:51.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.474+0000 7fc1d127e700 1 --2- 172.21.15.88:0/1393443585 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc1cc064d70 0x7fc1cc1632b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:51.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.474+0000 7fc1d0a7d700 1 --2- 172.21.15.88:0/1393443585 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc1cc1525f0 0x7fc1cc1637a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:51.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.474+0000 7fc1d1a7f700 1 -- 172.21.15.88:0/1393443585 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc1cc1525f0 msgr2=0x7fc1cc1637a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:51.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.474+0000 7fc1d1a7f700 1 --2- 172.21.15.88:0/1393443585 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc1cc1525f0 0x7fc1cc1637a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:51.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.474+0000 7fc1d1a7f700 1 -- 172.21.15.88:0/1393443585 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc1cc064d70 msgr2=0x7fc1cc1632b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:51.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.474+0000 7fc1d1a7f700 1 --2- 172.21.15.88:0/1393443585 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc1cc064d70 0x7fc1cc1632b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:51.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.474+0000 7fc1d1a7f700 1 -- 172.21.15.88:0/1393443585 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1c8009a20 con 0x7fc1cc153010 2023-10-18T16:08:51.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.474+0000 7fc1d127e700 1 --2- 172.21.15.88:0/1393443585 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc1cc064d70 0x7fc1cc1632b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:08:51.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.474+0000 7fc1d1a7f700 1 --2- 172.21.15.88:0/1393443585 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1cc153010 0x7fc1cc0ab110 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7fc1c4009e00 tx=0x7fc1c4017000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:08:51.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.475+0000 7fc1c17fa700 1 -- 172.21.15.88:0/1393443585 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c400fe60 con 0x7fc1cc153010 2023-10-18T16:08:51.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.475+0000 7fc1c17fa700 1 -- 172.21.15.88:0/1393443585 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1c401b410 con 0x7fc1cc153010 2023-10-18T16:08:51.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.475+0000 7fc1d34e2700 1 -- 172.21.15.88:0/1393443585 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1c40092a0 con 0x7fc1cc153010 2023-10-18T16:08:51.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.475+0000 7fc1c17fa700 1 -- 172.21.15.88:0/1393443585 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c401b7b0 con 0x7fc1cc153010 2023-10-18T16:08:51.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.475+0000 7fc1d34e2700 1 -- 172.21.15.88:0/1393443585 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1cc1516b0 con 0x7fc1cc153010 2023-10-18T16:08:51.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.476+0000 7fc1d34e2700 1 -- 172.21.15.88:0/1393443585 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1cc06c4e0 con 0x7fc1cc153010 2023-10-18T16:08:51.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.476+0000 7fc1c17fa700 1 -- 172.21.15.88:0/1393443585 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7fc1c401bba0 con 0x7fc1cc153010 2023-10-18T16:08:51.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.476+0000 7fc1c17fa700 1 --2- 172.21.15.88:0/1393443585 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fc1b005f150 0x7fc1b0061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:51.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.476+0000 7fc1c17fa700 1 -- 172.21.15.88:0/1393443585 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6467+0+0 (secure 0 0 0) 0x7fc1c407bec0 con 0x7fc1cc153010 2023-10-18T16:08:51.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.476+0000 7fc1d127e700 1 --2- 172.21.15.88:0/1393443585 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fc1b005f150 0x7fc1b0061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:51.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.477+0000 7fc1d127e700 1 --2- 172.21.15.88:0/1393443585 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fc1b005f150 0x7fc1b0061610 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fc1c80099f0 tx=0x7fc1c800b040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:08:51.492 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.480+0000 7fc1c17fa700 1 -- 172.21.15.88:0/1393443585 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7fc1c404d030 con 0x7fc1cc153010 2023-10-18T16:08:51.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.687+0000 7fc1d34e2700 1 -- 172.21.15.88:0/1393443585 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc1cc151390 con 0x7fc1cc153010 2023-10-18T16:08:51.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.688+0000 7fc1c17fa700 1 -- 172.21.15.88:0/1393443585 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7fc1c4050650 con 0x7fc1cc153010 2023-10-18T16:08:51.691 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:08:51.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.690+0000 7fc1d34e2700 1 -- 172.21.15.88:0/1393443585 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fc1b005f150 msgr2=0x7fc1b0061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:51.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.690+0000 7fc1d34e2700 1 --2- 172.21.15.88:0/1393443585 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fc1b005f150 0x7fc1b0061610 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fc1c80099f0 tx=0x7fc1c800b040).stop 2023-10-18T16:08:51.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.690+0000 7fc1d34e2700 1 -- 172.21.15.88:0/1393443585 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1cc153010 msgr2=0x7fc1cc0ab110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:51.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.690+0000 7fc1d34e2700 1 --2- 172.21.15.88:0/1393443585 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1cc153010 0x7fc1cc0ab110 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7fc1c4009e00 tx=0x7fc1c4017000).stop 2023-10-18T16:08:51.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.690+0000 7fc1d34e2700 1 -- 172.21.15.88:0/1393443585 shutdown_connections 2023-10-18T16:08:51.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.690+0000 7fc1d34e2700 1 --2- 172.21.15.88:0/1393443585 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fc1b005f150 0x7fc1b0061610 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:51.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.690+0000 7fc1d34e2700 1 --2- 172.21.15.88:0/1393443585 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1cc153010 0x7fc1cc0ab110 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:51.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.690+0000 7fc1d34e2700 1 --2- 172.21.15.88:0/1393443585 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc1cc064d70 0x7fc1cc1632b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:51.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.690+0000 7fc1d34e2700 1 --2- 172.21.15.88:0/1393443585 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc1cc1525f0 0x7fc1cc1637a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:51.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.690+0000 7fc1d34e2700 1 -- 172.21.15.88:0/1393443585 >> 172.21.15.88:0/1393443585 conn(0x7fc1cc069d80 msgr2=0x7fc1cc151ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:08:51.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.690+0000 7fc1d34e2700 1 -- 172.21.15.88:0/1393443585 shutdown_connections 2023-10-18T16:08:51.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:51.690+0000 7fc1d34e2700 1 -- 172.21.15.88:0/1393443585 wait complete. 2023-10-18T16:08:51.705 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:08:54.706 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:08:54.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:54.827+0000 7f02bec56700 1 Processor -- start 2023-10-18T16:08:54.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:54.827+0000 7f02bec56700 1 -- start start 2023-10-18T16:08:54.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:54.828+0000 7f02bec56700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f02b8065360 0x7f02b815bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:54.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:54.828+0000 7f02bec56700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02b8064150 con 0x7f02b814d8c0 2023-10-18T16:08:54.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:54.828+0000 7f02bec56700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02b8153a40 con 0x7f02b80a9050 2023-10-18T16:08:54.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:54.828+0000 7f02bec56700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02b8063e70 con 0x7f02b8065360 2023-10-18T16:08:54.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:54.828+0000 7f02bd1f3700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f02b8065360 0x7f02b815bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:54.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:54.828+0000 7f02bc9f2700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f02b814d8c0 0x7f02b8064440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:56988/0 (socket says 172.21.15.88:56988) 2023-10-18T16:08:54.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:54.828+0000 7f02bd1f3700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f02b8065360 0x7f02b815bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:39644/0 (socket says 172.21.15.88:39644) 2023-10-18T16:08:54.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:54.828+0000 7f02bc9f2700 1 -- 172.21.15.88:0/1548841800 learned_addr learned my addr 172.21.15.88:0/1548841800 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:08:54.835 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:54.829+0000 7f02bd1f3700 1 -- 172.21.15.88:0/1548841800 >> v1:172.21.15.151:6789/0 conn(0x7f02b80a9050 legacy=0x7f02b8153ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:08:54.835 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:54.829+0000 7f02bd1f3700 1 -- 172.21.15.88:0/1548841800 >> v1:172.21.15.88:6789/0 conn(0x7f02b814d8c0 legacy=0x7f02b8064440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:54.835 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:54.829+0000 7f02bd1f3700 1 -- 172.21.15.88:0/1548841800 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02b80c72c0 con 0x7f02b8065360 2023-10-18T16:08:54.835 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:54.829+0000 7f02bd1f3700 1 --2- 172.21.15.88:0/1548841800 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02b8065360 0x7f02b815bcc0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f02b0009760 tx=0x7f02b0009060).ready entity=mon.0 client_cookie=979b7b7ce3251030 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:08:54.835 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:54.829+0000 7f02af7fe700 1 -- 172.21.15.88:0/1548841800 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b00090d0 con 0x7f02b8065360 2023-10-18T16:08:54.836 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:54.829+0000 7f02af7fe700 1 -- 172.21.15.88:0/1548841800 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02b000f510 con 0x7f02b8065360 2023-10-18T16:08:54.836 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:54.829+0000 7f02af7fe700 1 -- 172.21.15.88:0/1548841800 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b000f890 con 0x7f02b8065360 2023-10-18T16:08:54.836 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:54.829+0000 7f02bec56700 1 -- 172.21.15.88:0/1548841800 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02b8065360 msgr2=0x7f02b815bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:54.836 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:54.829+0000 7f02bec56700 1 --2- 172.21.15.88:0/1548841800 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02b8065360 0x7f02b815bcc0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f02b0009760 tx=0x7f02b0009060).stop 2023-10-18T16:08:54.836 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:54.829+0000 7f02bec56700 1 -- 172.21.15.88:0/1548841800 shutdown_connections 2023-10-18T16:08:54.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:54.829+0000 7f02bec56700 1 --2- 172.21.15.88:0/1548841800 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02b8065360 0x7f02b815bcc0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:54.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:54.829+0000 7f02bec56700 1 -- 172.21.15.88:0/1548841800 >> 172.21.15.88:0/1548841800 conn(0x7f02b806a810 msgr2=0x7f02b80c6a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:08:54.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:54.830+0000 7f02bec56700 1 -- 172.21.15.88:0/1548841800 shutdown_connections 2023-10-18T16:08:54.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:54.830+0000 7f02bec56700 1 -- 172.21.15.88:0/1548841800 wait complete. 2023-10-18T16:08:54.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:54.831+0000 7f02bec56700 1 Processor -- start 2023-10-18T16:08:54.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:54.831+0000 7f02bec56700 1 -- start start 2023-10-18T16:08:54.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:54.831+0000 7f02bec56700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f02b8065360 0x7f02b8150960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:54.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:54.832+0000 7f02bec56700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02b80a9050 0x7f02b80abcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:54.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:54.832+0000 7f02bec56700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02b814d8c0 0x7f02b80bd0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:54.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:54.832+0000 7f02bec56700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b80c0610 con 0x7f02b814d8c0 2023-10-18T16:08:54.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:54.832+0000 7f02bec56700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b80ac1b0 con 0x7f02b80a9050 2023-10-18T16:08:54.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:54.832+0000 7f02bec56700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02b80acba0 con 0x7f02b8065360 2023-10-18T16:08:54.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:54.832+0000 7f02bd1f3700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02b814d8c0 0x7f02b80bd0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:54.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:54.832+0000 7f02bd1f3700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02b814d8c0 0x7f02b80bd0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:39660/0 (socket says 172.21.15.88:39660) 2023-10-18T16:08:54.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:54.832+0000 7f02bd1f3700 1 -- 172.21.15.88:0/1208023759 learned_addr learned my addr 172.21.15.88:0/1208023759 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:08:54.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:54.832+0000 7f02affff700 1 --2- 172.21.15.88:0/1208023759 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02b80a9050 0x7f02b80abcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:54.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:54.832+0000 7f02bd1f3700 1 -- 172.21.15.88:0/1208023759 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f02b8065360 msgr2=0x7f02b8150960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:54.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:54.832+0000 7f02bc9f2700 1 --2- 172.21.15.88:0/1208023759 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f02b8065360 0x7f02b8150960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:54.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:54.832+0000 7f02bd1f3700 1 --2- 172.21.15.88:0/1208023759 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f02b8065360 0x7f02b8150960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:54.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:54.832+0000 7f02bd1f3700 1 -- 172.21.15.88:0/1208023759 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02b80a9050 msgr2=0x7f02b80abcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:54.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:54.832+0000 7f02bd1f3700 1 --2- 172.21.15.88:0/1208023759 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02b80a9050 0x7f02b80abcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:54.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:54.832+0000 7f02bd1f3700 1 -- 172.21.15.88:0/1208023759 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02b00092a0 con 0x7f02b814d8c0 2023-10-18T16:08:54.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:54.832+0000 7f02affff700 1 --2- 172.21.15.88:0/1208023759 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02b80a9050 0x7f02b80abcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-10-18T16:08:54.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:54.832+0000 7f02bd1f3700 1 --2- 172.21.15.88:0/1208023759 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02b814d8c0 0x7f02b80bd0f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f02b8071c10 tx=0x7f02b0017000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:08:54.844 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:54.832+0000 7f02acff9700 1 -- 172.21.15.88:0/1208023759 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b000f890 con 0x7f02b814d8c0 2023-10-18T16:08:54.844 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:54.832+0000 7f02acff9700 1 -- 172.21.15.88:0/1208023759 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02b000fe10 con 0x7f02b814d8c0 2023-10-18T16:08:54.844 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:54.832+0000 7f02bec56700 1 -- 172.21.15.88:0/1208023759 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02b8160cc0 con 0x7f02b814d8c0 2023-10-18T16:08:54.844 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:54.832+0000 7f02acff9700 1 -- 172.21.15.88:0/1208023759 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b001b660 con 0x7f02b814d8c0 2023-10-18T16:08:54.844 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:54.832+0000 7f02bec56700 1 -- 172.21.15.88:0/1208023759 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02b81511b0 con 0x7f02b814d8c0 2023-10-18T16:08:54.844 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:54.833+0000 7f02acff9700 1 -- 172.21.15.88:0/1208023759 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f02b001ba50 con 0x7f02b814d8c0 2023-10-18T16:08:54.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:54.833+0000 7f02bec56700 1 -- 172.21.15.88:0/1208023759 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02b8160e50 con 0x7f02b814d8c0 2023-10-18T16:08:54.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:54.834+0000 7f02acff9700 1 --2- 172.21.15.88:0/1208023759 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f02a405f180 0x7f02a4061640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:54.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:54.834+0000 7f02acff9700 1 -- 172.21.15.88:0/1208023759 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6467+0+0 (secure 0 0 0) 0x7f02b007be80 con 0x7f02b814d8c0 2023-10-18T16:08:54.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:54.834+0000 7f02bc9f2700 1 --2- 172.21.15.88:0/1208023759 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f02a405f180 0x7f02a4061640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:54.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:54.834+0000 7f02bc9f2700 1 --2- 172.21.15.88:0/1208023759 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f02a405f180 0x7f02a4061640 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f02b4001460 tx=0x7f02b4009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:08:54.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:54.838+0000 7f02acff9700 1 -- 172.21.15.88:0/1208023759 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f02b004cff0 con 0x7f02b814d8c0 2023-10-18T16:08:55.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:55.039+0000 7f02bec56700 1 -- 172.21.15.88:0/1208023759 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f02b80c75a0 con 0x7f02b814d8c0 2023-10-18T16:08:55.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:55.039+0000 7f02acff9700 1 -- 172.21.15.88:0/1208023759 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7f02b0050610 con 0x7f02b814d8c0 2023-10-18T16:08:55.043 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:08:55.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:55.041+0000 7f02bec56700 1 -- 172.21.15.88:0/1208023759 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f02a405f180 msgr2=0x7f02a4061640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:55.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:55.041+0000 7f02bec56700 1 --2- 172.21.15.88:0/1208023759 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f02a405f180 0x7f02a4061640 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f02b4001460 tx=0x7f02b4009040).stop 2023-10-18T16:08:55.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:55.041+0000 7f02bec56700 1 -- 172.21.15.88:0/1208023759 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02b814d8c0 msgr2=0x7f02b80bd0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:55.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:55.041+0000 7f02bec56700 1 --2- 172.21.15.88:0/1208023759 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02b814d8c0 0x7f02b80bd0f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f02b8071c10 tx=0x7f02b0017000).stop 2023-10-18T16:08:55.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:55.041+0000 7f02bec56700 1 -- 172.21.15.88:0/1208023759 shutdown_connections 2023-10-18T16:08:55.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:55.041+0000 7f02bec56700 1 --2- 172.21.15.88:0/1208023759 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f02a405f180 0x7f02a4061640 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:55.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:55.041+0000 7f02bec56700 1 --2- 172.21.15.88:0/1208023759 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02b814d8c0 0x7f02b80bd0f0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:55.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:55.041+0000 7f02bec56700 1 --2- 172.21.15.88:0/1208023759 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f02b8065360 0x7f02b8150960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:55.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:55.041+0000 7f02bec56700 1 --2- 172.21.15.88:0/1208023759 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02b80a9050 0x7f02b80abcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:55.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:55.041+0000 7f02bec56700 1 -- 172.21.15.88:0/1208023759 >> 172.21.15.88:0/1208023759 conn(0x7f02b806a810 msgr2=0x7f02b814c110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:08:55.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:55.041+0000 7f02bec56700 1 -- 172.21.15.88:0/1208023759 shutdown_connections 2023-10-18T16:08:55.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:55.041+0000 7f02bec56700 1 -- 172.21.15.88:0/1208023759 wait complete. 2023-10-18T16:08:55.057 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:08:57.107 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~7s 2023-10-18T16:08:57.107 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~68s 2023-10-18T16:08:57.107 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.f is failed for ~20s 2023-10-18T16:08:58.058 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:08:58.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.179+0000 7f1049a4d700 1 Processor -- start 2023-10-18T16:08:58.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.179+0000 7f1049a4d700 1 -- start start 2023-10-18T16:08:58.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.180+0000 7f1049a4d700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f1044068710 0x7f1044068af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:58.191 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.180+0000 7f1049a4d700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1044064130 0x7f1044150150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:58.191 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.180+0000 7f1049a4d700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1044066920 con 0x7f10440678d0 2023-10-18T16:08:58.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.180+0000 7f1049a4d700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10440674c0 con 0x7f1044068710 2023-10-18T16:08:58.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.180+0000 7f1049a4d700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1044150720 con 0x7f1044064130 2023-10-18T16:08:58.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.180+0000 7f104924c700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f10440678d0 0x7f1044066db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:33546/0 (socket says 172.21.15.88:33546) 2023-10-18T16:08:58.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.180+0000 7f104924c700 1 -- 172.21.15.88:0/2208804032 learned_addr learned my addr 172.21.15.88:0/2208804032 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:08:58.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.180+0000 7f103b7fe700 1 -- 172.21.15.88:0/2208804032 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1785825722 0 0) 0x7f1044066920 con 0x7f10440678d0 2023-10-18T16:08:58.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.180+0000 7f103b7fe700 1 -- 172.21.15.88:0/2208804032 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f102c0033f0 con 0x7f10440678d0 2023-10-18T16:08:58.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.180+0000 7f1048a4b700 1 --2- 172.21.15.88:0/2208804032 >> v2:172.21.15.151:3300/0 conn(0x7f1044068710 0x7f1044068af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:58.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.180+0000 7f103bfff700 1 --2- 172.21.15.88:0/2208804032 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1044064130 0x7f1044150150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:58.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.181+0000 7f103b7fe700 1 -- 172.21.15.88:0/2208804032 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2588491646 0 0) 0x7f102c0033f0 con 0x7f10440678d0 2023-10-18T16:08:58.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.181+0000 7f103b7fe700 1 -- 172.21.15.88:0/2208804032 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1044064130 msgr2=0x7f1044150150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:08:58.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.181+0000 7f103b7fe700 1 --2- 172.21.15.88:0/2208804032 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1044064130 0x7f1044150150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:58.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.181+0000 7f103b7fe700 1 -- 172.21.15.88:0/2208804032 >> v2:172.21.15.151:3300/0 conn(0x7f1044068710 msgr2=0x7f1044068af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:08:58.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.181+0000 7f103b7fe700 1 --2- 172.21.15.88:0/2208804032 >> v2:172.21.15.151:3300/0 conn(0x7f1044068710 0x7f1044068af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:58.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.181+0000 7f103b7fe700 1 -- 172.21.15.88:0/2208804032 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1044150a00 con 0x7f10440678d0 2023-10-18T16:08:58.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.181+0000 7f103b7fe700 1 -- 172.21.15.88:0/2208804032 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441715075 0 0) 0x7f103c0017f0 con 0x7f10440678d0 2023-10-18T16:08:58.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.181+0000 7f103b7fe700 1 -- 172.21.15.88:0/2208804032 >> v1:172.21.15.88:6789/0 conn(0x7f10440678d0 legacy=0x7f1044066db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:58.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.181+0000 7f103b7fe700 1 --2- 172.21.15.88:0/2208804032 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f102c0038e0 0x7f102c003d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:58.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.181+0000 7f103b7fe700 1 --2- 172.21.15.88:0/2208804032 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f102c004690 0x7f102c004b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:58.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.181+0000 7f103b7fe700 1 --2- 172.21.15.88:0/2208804032 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1044064130 0x7f102c005000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:58.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.181+0000 7f103b7fe700 1 -- 172.21.15.88:0/2208804032 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10440674c0 con 0x7f102c004690 2023-10-18T16:08:58.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.181+0000 7f103b7fe700 1 -- 172.21.15.88:0/2208804032 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1044150720 con 0x7f102c0038e0 2023-10-18T16:08:58.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.181+0000 7f103b7fe700 1 -- 172.21.15.88:0/2208804032 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f102c00b8e0 con 0x7f1044064130 2023-10-18T16:08:58.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.181+0000 7f103bfff700 1 --2- 172.21.15.88:0/2208804032 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f102c004690 0x7f102c004b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:58.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.181+0000 7f1048a4b700 1 --2- 172.21.15.88:0/2208804032 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f102c0038e0 0x7f102c003d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:58.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.181+0000 7f104924c700 1 --2- 172.21.15.88:0/2208804032 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1044064130 0x7f102c005000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:58.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.181+0000 7f103bfff700 1 -- 172.21.15.88:0/2208804032 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1044064130 msgr2=0x7f102c005000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:08:58.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.181+0000 7f103bfff700 1 --2- 172.21.15.88:0/2208804032 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1044064130 0x7f102c005000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:58.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.181+0000 7f103bfff700 1 -- 172.21.15.88:0/2208804032 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f102c0038e0 msgr2=0x7f102c003d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:08:58.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.181+0000 7f103bfff700 1 --2- 172.21.15.88:0/2208804032 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f102c0038e0 0x7f102c003d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:58.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.181+0000 7f103bfff700 1 -- 172.21.15.88:0/2208804032 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f102c0033f0 con 0x7f102c004690 2023-10-18T16:08:58.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.181+0000 7f103bfff700 1 --2- 172.21.15.88:0/2208804032 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f102c004690 0x7f102c004b10 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f1034009a60 tx=0x7f1034009060).ready entity=mon.0 client_cookie=c5d81381427401b7 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:08:58.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.181+0000 7f103b7fe700 1 -- 172.21.15.88:0/2208804032 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1034009ea0 con 0x7f102c004690 2023-10-18T16:08:58.198 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.181+0000 7f103b7fe700 1 -- 172.21.15.88:0/2208804032 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1034009430 con 0x7f102c004690 2023-10-18T16:08:58.198 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.181+0000 7f103b7fe700 1 -- 172.21.15.88:0/2208804032 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f103400e870 con 0x7f102c004690 2023-10-18T16:08:58.198 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.182+0000 7f1049a4d700 1 -- 172.21.15.88:0/2208804032 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f102c004690 msgr2=0x7f102c004b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:58.198 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.182+0000 7f1049a4d700 1 --2- 172.21.15.88:0/2208804032 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f102c004690 0x7f102c004b10 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f1034009a60 tx=0x7f1034009060).stop 2023-10-18T16:08:58.198 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.182+0000 7f104924c700 1 -- 172.21.15.88:0/2208804032 reap_dead start 2023-10-18T16:08:58.199 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.182+0000 7f1049a4d700 1 -- 172.21.15.88:0/2208804032 shutdown_connections 2023-10-18T16:08:58.199 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.182+0000 7f1049a4d700 1 -- 172.21.15.88:0/2208804032 >> 172.21.15.88:0/2208804032 conn(0x7f10440c4770 msgr2=0x7f10440c4b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:08:58.199 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.182+0000 7f1049a4d700 1 -- 172.21.15.88:0/2208804032 shutdown_connections 2023-10-18T16:08:58.199 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.182+0000 7f1049a4d700 1 -- 172.21.15.88:0/2208804032 wait complete. 2023-10-18T16:08:58.199 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.183+0000 7f1049a4d700 1 Processor -- start 2023-10-18T16:08:58.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.183+0000 7f1049a4d700 1 -- start start 2023-10-18T16:08:58.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.184+0000 7f1049a4d700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f102c004690 0x7f1044151d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:58.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.184+0000 7f1049a4d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1044152260 0x7f1044153260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:58.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.184+0000 7f1049a4d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1044154130 0x7f1044153750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:58.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.184+0000 7f1049a4d700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1044152dd0 con 0x7f1044154130 2023-10-18T16:08:58.201 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.184+0000 7f1049a4d700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1044153c40 con 0x7f1044152260 2023-10-18T16:08:58.201 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.184+0000 7f1049a4d700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10440be760 con 0x7f102c004690 2023-10-18T16:08:58.201 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.184+0000 7f104924c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1044154130 0x7f1044153750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:58.201 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.184+0000 7f104924c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1044154130 0x7f1044153750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35674/0 (socket says 172.21.15.88:35674) 2023-10-18T16:08:58.201 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.184+0000 7f104924c700 1 -- 172.21.15.88:0/585791214 learned_addr learned my addr 172.21.15.88:0/585791214 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:08:58.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.184+0000 7f103bfff700 1 --2- 172.21.15.88:0/585791214 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1044152260 0x7f1044153260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:58.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.184+0000 7f1048a4b700 1 --2- 172.21.15.88:0/585791214 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f102c004690 0x7f1044151d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:58.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.184+0000 7f104924c700 1 -- 172.21.15.88:0/585791214 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f102c004690 msgr2=0x7f1044151d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:58.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.184+0000 7f104924c700 1 --2- 172.21.15.88:0/585791214 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f102c004690 0x7f1044151d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:58.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.184+0000 7f104924c700 1 -- 172.21.15.88:0/585791214 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1044152260 msgr2=0x7f1044153260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:58.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.184+0000 7f104924c700 1 --2- 172.21.15.88:0/585791214 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1044152260 0x7f1044153260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:58.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.184+0000 7f104924c700 1 -- 172.21.15.88:0/585791214 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10440bdec0 con 0x7f1044154130 2023-10-18T16:08:58.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.184+0000 7f103bfff700 1 --2- 172.21.15.88:0/585791214 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1044152260 0x7f1044153260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:08:58.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.184+0000 7f104924c700 1 --2- 172.21.15.88:0/585791214 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1044154130 0x7f1044153750 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f103c000f80 tx=0x7f103c001d90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:08:58.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.185+0000 7f1038ff9700 1 -- 172.21.15.88:0/585791214 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f103c0020a0 con 0x7f1044154130 2023-10-18T16:08:58.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.185+0000 7f1038ff9700 1 -- 172.21.15.88:0/585791214 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f103c00fbb0 con 0x7f1044154130 2023-10-18T16:08:58.204 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.185+0000 7f1038ff9700 1 -- 172.21.15.88:0/585791214 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f103c012350 con 0x7f1044154130 2023-10-18T16:08:58.204 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.185+0000 7f1049a4d700 1 -- 172.21.15.88:0/585791214 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10440be1a0 con 0x7f1044154130 2023-10-18T16:08:58.204 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.185+0000 7f1049a4d700 1 -- 172.21.15.88:0/585791214 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1044152a60 con 0x7f1044154130 2023-10-18T16:08:58.204 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.186+0000 7f1049a4d700 1 -- 172.21.15.88:0/585791214 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1044152f80 con 0x7f1044154130 2023-10-18T16:08:58.204 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.186+0000 7f1038ff9700 1 -- 172.21.15.88:0/585791214 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f103c0126c0 con 0x7f1044154130 2023-10-18T16:08:58.204 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.186+0000 7f1038ff9700 1 --2- 172.21.15.88:0/585791214 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f103005f150 0x7f1030061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:08:58.205 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.186+0000 7f1038ff9700 1 -- 172.21.15.88:0/585791214 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6467+0+0 (secure 0 0 0) 0x7f103c079d20 con 0x7f1044154130 2023-10-18T16:08:58.205 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.186+0000 7f1048a4b700 1 --2- 172.21.15.88:0/585791214 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f103005f150 0x7f1030061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:08:58.205 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.187+0000 7f1048a4b700 1 --2- 172.21.15.88:0/585791214 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f103005f150 0x7f1030061610 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f1040006d80 tx=0x7f1040008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:08:58.205 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.190+0000 7f1038ff9700 1 -- 172.21.15.88:0/585791214 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f1044152f80 con 0x7f1044154130 2023-10-18T16:08:58.394 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.390+0000 7f1049a4d700 1 -- 172.21.15.88:0/585791214 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f10440bd600 con 0x7f1044154130 2023-10-18T16:08:58.395 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.391+0000 7f1038ff9700 1 -- 172.21.15.88:0/585791214 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7f10440bd600 con 0x7f1044154130 2023-10-18T16:08:58.395 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:08:58.396 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.392+0000 7f1049a4d700 1 -- 172.21.15.88:0/585791214 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f103005f150 msgr2=0x7f1030061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:58.397 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.392+0000 7f1049a4d700 1 --2- 172.21.15.88:0/585791214 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f103005f150 0x7f1030061610 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f1040006d80 tx=0x7f1040008040).stop 2023-10-18T16:08:58.397 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.393+0000 7f1049a4d700 1 -- 172.21.15.88:0/585791214 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1044154130 msgr2=0x7f1044153750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:08:58.397 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.393+0000 7f1049a4d700 1 --2- 172.21.15.88:0/585791214 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1044154130 0x7f1044153750 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f103c000f80 tx=0x7f103c001d90).stop 2023-10-18T16:08:58.397 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.393+0000 7f1049a4d700 1 -- 172.21.15.88:0/585791214 shutdown_connections 2023-10-18T16:08:58.397 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.393+0000 7f1049a4d700 1 --2- 172.21.15.88:0/585791214 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f103005f150 0x7f1030061610 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:58.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.393+0000 7f1049a4d700 1 --2- 172.21.15.88:0/585791214 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1044154130 0x7f1044153750 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:58.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.393+0000 7f1049a4d700 1 --2- 172.21.15.88:0/585791214 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f102c004690 0x7f1044151d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:58.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.393+0000 7f1049a4d700 1 --2- 172.21.15.88:0/585791214 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1044152260 0x7f1044153260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:08:58.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.393+0000 7f1049a4d700 1 -- 172.21.15.88:0/585791214 >> 172.21.15.88:0/585791214 conn(0x7f10440c4770 msgr2=0x7f1044063760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:08:58.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.393+0000 7f1049a4d700 1 -- 172.21.15.88:0/585791214 shutdown_connections 2023-10-18T16:08:58.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:08:58.393+0000 7f1049a4d700 1 -- 172.21.15.88:0/585791214 wait complete. 2023-10-18T16:08:58.408 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:09:01.409 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:09:01.535 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.532+0000 7fdc55e76700 1 Processor -- start 2023-10-18T16:09:01.536 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.532+0000 7fdc55e76700 1 -- start start 2023-10-18T16:09:01.538 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.533+0000 7fdc55e76700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fdc5014d8c0 0x7fdc50064440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:01.538 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.533+0000 7fdc55e76700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdc500a9050 0x7fdc50064930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:01.538 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.533+0000 7fdc55e76700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fdc50065360 0x7fdc50153ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:01.538 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.533+0000 7fdc55e76700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc50064150 con 0x7fdc5014d8c0 2023-10-18T16:09:01.539 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.533+0000 7fdc55e76700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc50153a40 con 0x7fdc50065360 2023-10-18T16:09:01.539 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.533+0000 7fdc55e76700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc50063e70 con 0x7fdc500a9050 2023-10-18T16:09:01.539 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.533+0000 7fdc4f7fe700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fdc5014d8c0 0x7fdc50064440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:01.539 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.533+0000 7fdc4f7fe700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fdc5014d8c0 0x7fdc50064440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35686/0 (socket says 172.21.15.88:35686) 2023-10-18T16:09:01.539 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.533+0000 7fdc4f7fe700 1 -- 172.21.15.88:0/133080188 learned_addr learned my addr 172.21.15.88:0/133080188 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:09:01.540 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.533+0000 7fdc4ffff700 1 --2- 172.21.15.88:0/133080188 >> v2:172.21.15.151:3300/0 conn(0x7fdc50065360 0x7fdc50153ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:01.540 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.533+0000 7fdc4effd700 1 --2- 172.21.15.88:0/133080188 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdc500a9050 0x7fdc50064930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:01.540 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.533+0000 7fdc4f7fe700 1 -- 172.21.15.88:0/133080188 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdc500a9050 msgr2=0x7fdc50064930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:09:01.540 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.533+0000 7fdc4f7fe700 1 --2- 172.21.15.88:0/133080188 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdc500a9050 0x7fdc50064930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:01.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.533+0000 7fdc4f7fe700 1 -- 172.21.15.88:0/133080188 >> v2:172.21.15.151:3300/0 conn(0x7fdc50065360 msgr2=0x7fdc50153ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:09:01.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.533+0000 7fdc4f7fe700 1 --2- 172.21.15.88:0/133080188 >> v2:172.21.15.151:3300/0 conn(0x7fdc50065360 0x7fdc50153ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:01.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.533+0000 7fdc4f7fe700 1 -- 172.21.15.88:0/133080188 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc500c72c0 con 0x7fdc5014d8c0 2023-10-18T16:09:01.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.533+0000 7fdc4ffff700 1 --2- 172.21.15.88:0/133080188 >> v2:172.21.15.151:3300/0 conn(0x7fdc50065360 0x7fdc50153ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:09:01.542 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.533+0000 7fdc4f7fe700 1 --2- 172.21.15.88:0/133080188 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdc5014d8c0 0x7fdc50064440 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7fdc500ba710 tx=0x7fdc440090f0).ready entity=mon.0 client_cookie=5b16760ad2e72436 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:09:01.542 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.533+0000 7fdc4e7fc700 1 -- 172.21.15.88:0/133080188 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc4400b600 con 0x7fdc5014d8c0 2023-10-18T16:09:01.542 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.533+0000 7fdc4e7fc700 1 -- 172.21.15.88:0/133080188 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc4400b760 con 0x7fdc5014d8c0 2023-10-18T16:09:01.542 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.533+0000 7fdc4e7fc700 1 -- 172.21.15.88:0/133080188 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc4400bb00 con 0x7fdc5014d8c0 2023-10-18T16:09:01.542 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.534+0000 7fdc55e76700 1 -- 172.21.15.88:0/133080188 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdc5014d8c0 msgr2=0x7fdc50064440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:01.542 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.534+0000 7fdc55e76700 1 --2- 172.21.15.88:0/133080188 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdc5014d8c0 0x7fdc50064440 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7fdc500ba710 tx=0x7fdc440090f0).stop 2023-10-18T16:09:01.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.534+0000 7fdc55e76700 1 -- 172.21.15.88:0/133080188 shutdown_connections 2023-10-18T16:09:01.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.534+0000 7fdc55e76700 1 --2- 172.21.15.88:0/133080188 >> v2:172.21.15.151:3300/0 conn(0x7fdc50065360 0x7fdc50153ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:01.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.534+0000 7fdc55e76700 1 --2- 172.21.15.88:0/133080188 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdc5014d8c0 0x7fdc50064440 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:01.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.534+0000 7fdc55e76700 1 --2- 172.21.15.88:0/133080188 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdc500a9050 0x7fdc50064930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:01.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.534+0000 7fdc55e76700 1 -- 172.21.15.88:0/133080188 >> 172.21.15.88:0/133080188 conn(0x7fdc5006a810 msgr2=0x7fdc500c6a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:09:01.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.534+0000 7fdc55e76700 1 -- 172.21.15.88:0/133080188 shutdown_connections 2023-10-18T16:09:01.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.534+0000 7fdc55e76700 1 -- 172.21.15.88:0/133080188 wait complete. 2023-10-18T16:09:01.547 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.535+0000 7fdc55e76700 1 Processor -- start 2023-10-18T16:09:01.547 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.535+0000 7fdc55e76700 1 -- start start 2023-10-18T16:09:01.547 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.536+0000 7fdc55e76700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdc50065360 0x7fdc500bbc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:01.547 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.536+0000 7fdc55e76700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdc500a9050 0x7fdc500bc180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:01.548 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.536+0000 7fdc55e76700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdc5014d8c0 0x7fdc50148bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:01.548 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.536+0000 7fdc55e76700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc500bb870 con 0x7fdc50065360 2023-10-18T16:09:01.548 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.536+0000 7fdc55e76700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc500c6b60 con 0x7fdc5014d8c0 2023-10-18T16:09:01.548 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.536+0000 7fdc55e76700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc500ab570 con 0x7fdc500a9050 2023-10-18T16:09:01.548 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.536+0000 7fdc4f7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdc50065360 0x7fdc500bbc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:01.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.536+0000 7fdc4f7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdc50065360 0x7fdc500bbc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35692/0 (socket says 172.21.15.88:35692) 2023-10-18T16:09:01.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.536+0000 7fdc4f7fe700 1 -- 172.21.15.88:0/1528121526 learned_addr learned my addr 172.21.15.88:0/1528121526 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:09:01.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.537+0000 7fdc4f7fe700 1 -- 172.21.15.88:0/1528121526 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdc500a9050 msgr2=0x7fdc500bc180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:01.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.537+0000 7fdc4f7fe700 1 --2- 172.21.15.88:0/1528121526 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdc500a9050 0x7fdc500bc180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-10-18T16:09:01.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.537+0000 7fdc4f7fe700 1 -- 172.21.15.88:0/1528121526 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdc5014d8c0 msgr2=0x7fdc50148bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:01.550 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.537+0000 7fdc4f7fe700 1 --2- 172.21.15.88:0/1528121526 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdc5014d8c0 0x7fdc50148bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-10-18T16:09:01.550 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.537+0000 7fdc4f7fe700 1 -- 172.21.15.88:0/1528121526 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc44009370 con 0x7fdc50065360 2023-10-18T16:09:01.550 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.537+0000 7fdc4f7fe700 1 --2- 172.21.15.88:0/1528121526 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdc50065360 0x7fdc500bbc90 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7fdc50071c10 tx=0x7fdc44017000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:09:01.550 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.537+0000 7fdc377fe700 1 -- 172.21.15.88:0/1528121526 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc44002f30 con 0x7fdc50065360 2023-10-18T16:09:01.550 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.537+0000 7fdc377fe700 1 -- 172.21.15.88:0/1528121526 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc4400b8d0 con 0x7fdc50065360 2023-10-18T16:09:01.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.537+0000 7fdc55e76700 1 -- 172.21.15.88:0/1528121526 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc500ab850 con 0x7fdc50065360 2023-10-18T16:09:01.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.537+0000 7fdc377fe700 1 -- 172.21.15.88:0/1528121526 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc44024470 con 0x7fdc50065360 2023-10-18T16:09:01.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.537+0000 7fdc55e76700 1 -- 172.21.15.88:0/1528121526 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc500abe10 con 0x7fdc50065360 2023-10-18T16:09:01.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.538+0000 7fdc55e76700 1 -- 172.21.15.88:0/1528121526 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc500ab340 con 0x7fdc50065360 2023-10-18T16:09:01.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.538+0000 7fdc377fe700 1 -- 172.21.15.88:0/1528121526 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7fdc44021070 con 0x7fdc50065360 2023-10-18T16:09:01.552 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.538+0000 7fdc377fe700 1 --2- 172.21.15.88:0/1528121526 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fdc3c05f150 0x7fdc3c061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:01.552 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.538+0000 7fdc377fe700 1 -- 172.21.15.88:0/1528121526 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6467+0+0 (secure 0 0 0) 0x7fdc4407bd40 con 0x7fdc50065360 2023-10-18T16:09:01.552 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.539+0000 7fdc4effd700 1 --2- 172.21.15.88:0/1528121526 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fdc3c05f150 0x7fdc3c061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:01.552 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.539+0000 7fdc4effd700 1 --2- 172.21.15.88:0/1528121526 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fdc3c05f150 0x7fdc3c061610 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7fdc500c3490 tx=0x7fdc40008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:09:01.552 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.543+0000 7fdc377fe700 1 -- 172.21.15.88:0/1528121526 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7fdc4404ceb0 con 0x7fdc50065360 2023-10-18T16:09:01.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.745+0000 7fdc55e76700 1 -- 172.21.15.88:0/1528121526 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fdc5006c4e0 con 0x7fdc50065360 2023-10-18T16:09:01.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.745+0000 7fdc377fe700 1 -- 172.21.15.88:0/1528121526 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7fdc440504d0 con 0x7fdc50065360 2023-10-18T16:09:01.749 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:09:01.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.747+0000 7fdc55e76700 1 -- 172.21.15.88:0/1528121526 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fdc3c05f150 msgr2=0x7fdc3c061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:01.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.747+0000 7fdc55e76700 1 --2- 172.21.15.88:0/1528121526 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fdc3c05f150 0x7fdc3c061610 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7fdc500c3490 tx=0x7fdc40008040).stop 2023-10-18T16:09:01.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.747+0000 7fdc55e76700 1 -- 172.21.15.88:0/1528121526 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdc50065360 msgr2=0x7fdc500bbc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:01.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.747+0000 7fdc55e76700 1 --2- 172.21.15.88:0/1528121526 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdc50065360 0x7fdc500bbc90 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7fdc50071c10 tx=0x7fdc44017000).stop 2023-10-18T16:09:01.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.747+0000 7fdc55e76700 1 -- 172.21.15.88:0/1528121526 shutdown_connections 2023-10-18T16:09:01.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.747+0000 7fdc55e76700 1 --2- 172.21.15.88:0/1528121526 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fdc3c05f150 0x7fdc3c061610 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:01.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.747+0000 7fdc55e76700 1 --2- 172.21.15.88:0/1528121526 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdc5014d8c0 0x7fdc50148bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-10-18T16:09:01.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.747+0000 7fdc55e76700 1 --2- 172.21.15.88:0/1528121526 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdc50065360 0x7fdc500bbc90 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:01.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.747+0000 7fdc55e76700 1 --2- 172.21.15.88:0/1528121526 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdc500a9050 0x7fdc500bc180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-10-18T16:09:01.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.747+0000 7fdc55e76700 1 -- 172.21.15.88:0/1528121526 >> 172.21.15.88:0/1528121526 conn(0x7fdc5006a810 msgr2=0x7fdc50153ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:09:01.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.747+0000 7fdc55e76700 1 -- 172.21.15.88:0/1528121526 shutdown_connections 2023-10-18T16:09:01.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:01.747+0000 7fdc55e76700 1 -- 172.21.15.88:0/1528121526 wait complete. 2023-10-18T16:09:01.762 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:09:03.428 INFO:tasks.ceph.mds.b.smithi151.stderr:terminate called after throwing an instance of 'boost::wrapexcept' 2023-10-18T16:09:03.428 INFO:tasks.ceph.mds.b.smithi151.stderr: what(): boost::bad_get: failed value get using boost::get 2023-10-18T16:09:03.428 INFO:tasks.ceph.mds.b.smithi151.stderr:*** Caught signal (Aborted) ** 2023-10-18T16:09:03.429 INFO:tasks.ceph.mds.b.smithi151.stderr: in thread 7f2253fb0700 thread_name:ms_dispatch 2023-10-18T16:09:03.429 INFO:tasks.ceph.mds.b.smithi151.stderr: ceph version 16.2.14-100-gb4ddd52c (b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5) pacific (stable) 2023-10-18T16:09:03.429 INFO:tasks.ceph.mds.b.smithi151.stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f225b7b3b20] 2023-10-18T16:09:03.429 INFO:tasks.ceph.mds.b.smithi151.stderr: 2: gsignal() 2023-10-18T16:09:03.430 INFO:tasks.ceph.mds.b.smithi151.stderr: 3: abort() 2023-10-18T16:09:03.430 INFO:tasks.ceph.mds.b.smithi151.stderr: 4: /lib64/libstdc++.so.6(+0x9009b) [0x7f225abb409b] 2023-10-18T16:09:03.430 INFO:tasks.ceph.mds.b.smithi151.stderr: 5: /lib64/libstdc++.so.6(+0x9653c) [0x7f225abba53c] 2023-10-18T16:09:03.430 INFO:tasks.ceph.mds.b.smithi151.stderr: 6: /lib64/libstdc++.so.6(+0x96597) [0x7f225abba597] 2023-10-18T16:09:03.430 INFO:tasks.ceph.mds.b.smithi151.stderr: 7: /lib64/libstdc++.so.6(+0x967f8) [0x7f225abba7f8] 2023-10-18T16:09:03.431 INFO:tasks.ceph.mds.b.smithi151.stderr: 8: ceph-mds(+0x212b43) [0x5555b3b4db43] 2023-10-18T16:09:03.431 INFO:tasks.ceph.mds.b.smithi151.stderr: 9: (Option::size_t const md_config_t::get_val(ConfigValues const&, std::basic_string_view >) const+0x5c) [0x5555b3bc289c] 2023-10-18T16:09:03.431 INFO:tasks.ceph.mds.b.smithi151.stderr: 10: (SessionMap::SessionMap(MDSRank*)+0x2c2) [0x5555b3e0a6a2] 2023-10-18T16:09:03.431 INFO:tasks.ceph.mds.b.smithi151.stderr: 11: (MDSRank::MDSRank(int, std::__cxx11::basic_string, std::allocator >, ceph::fair_mutex&, std::shared_ptr&, CommonSafeTimer&, Beacon&, std::unique_ptr >&, Messenger*, MonClient*, MgrClient*, Context*, Context*, boost::asio::io_context&)+0x214) [0x5555b3b27724] 2023-10-18T16:09:03.431 INFO:tasks.ceph.mds.b.smithi151.stderr: 12: (MDSRankDispatcher::MDSRankDispatcher(int, std::__cxx11::basic_string, std::allocator >, ceph::fair_mutex&, std::shared_ptr&, CommonSafeTimer&, Beacon&, std::unique_ptr >&, Messenger*, MonClient*, MgrClient*, Context*, Context*, boost::asio::io_context&)+0x101) [0x5555b3b29181] 2023-10-18T16:09:03.432 INFO:tasks.ceph.mds.b.smithi151.stderr: 13: (MDSDaemon::handle_mds_map(boost::intrusive_ptr const&)+0x1608) [0x5555b3af9348] 2023-10-18T16:09:03.432 INFO:tasks.ceph.mds.b.smithi151.stderr: 14: (MDSDaemon::handle_core_message(boost::intrusive_ptr const&)+0xcd) [0x5555b3afc44d] 2023-10-18T16:09:03.432 INFO:tasks.ceph.mds.b.smithi151.stderr: 15: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0xc3) [0x5555b3afcfa3] 2023-10-18T16:09:03.432 INFO:tasks.ceph.mds.b.smithi151.stderr: 16: (DispatchQueue::entry()+0x126a) [0x7f225ca0d4ea] 2023-10-18T16:09:03.432 INFO:tasks.ceph.mds.b.smithi151.stderr: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f225cac1d51] 2023-10-18T16:09:03.432 INFO:tasks.ceph.mds.b.smithi151.stderr: 18: /lib64/libpthread.so.0(+0x814a) [0x7f225b7a914a] 2023-10-18T16:09:03.433 INFO:tasks.ceph.mds.b.smithi151.stderr: 19: clone() 2023-10-18T16:09:03.433 INFO:tasks.ceph.mds.b.smithi151.stderr:2023-10-18T16:09:03.426+0000 7f2253fb0700 -1 *** Caught signal (Aborted) ** 2023-10-18T16:09:03.433 INFO:tasks.ceph.mds.b.smithi151.stderr: in thread 7f2253fb0700 thread_name:ms_dispatch 2023-10-18T16:09:03.433 INFO:tasks.ceph.mds.b.smithi151.stderr: 2023-10-18T16:09:03.433 INFO:tasks.ceph.mds.b.smithi151.stderr: ceph version 16.2.14-100-gb4ddd52c (b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5) pacific (stable) 2023-10-18T16:09:03.434 INFO:tasks.ceph.mds.b.smithi151.stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f225b7b3b20] 2023-10-18T16:09:03.434 INFO:tasks.ceph.mds.b.smithi151.stderr: 2: gsignal() 2023-10-18T16:09:03.434 INFO:tasks.ceph.mds.b.smithi151.stderr: 3: abort() 2023-10-18T16:09:03.434 INFO:tasks.ceph.mds.b.smithi151.stderr: 4: /lib64/libstdc++.so.6(+0x9009b) [0x7f225abb409b] 2023-10-18T16:09:03.434 INFO:tasks.ceph.mds.b.smithi151.stderr: 5: /lib64/libstdc++.so.6(+0x9653c) [0x7f225abba53c] 2023-10-18T16:09:03.434 INFO:tasks.ceph.mds.b.smithi151.stderr: 6: /lib64/libstdc++.so.6(+0x96597) [0x7f225abba597] 2023-10-18T16:09:03.435 INFO:tasks.ceph.mds.b.smithi151.stderr: 7: /lib64/libstdc++.so.6(+0x967f8) [0x7f225abba7f8] 2023-10-18T16:09:03.435 INFO:tasks.ceph.mds.b.smithi151.stderr: 8: ceph-mds(+0x212b43) [0x5555b3b4db43] 2023-10-18T16:09:03.435 INFO:tasks.ceph.mds.b.smithi151.stderr: 9: (Option::size_t const md_config_t::get_val(ConfigValues const&, std::basic_string_view >) const+0x5c) [0x5555b3bc289c] 2023-10-18T16:09:03.435 INFO:tasks.ceph.mds.b.smithi151.stderr: 10: (SessionMap::SessionMap(MDSRank*)+0x2c2) [0x5555b3e0a6a2] 2023-10-18T16:09:03.435 INFO:tasks.ceph.mds.b.smithi151.stderr: 11: (MDSRank::MDSRank(int, std::__cxx11::basic_string, std::allocator >, ceph::fair_mutex&, std::shared_ptr&, CommonSafeTimer&, Beacon&, std::unique_ptr >&, Messenger*, MonClient*, MgrClient*, Context*, Context*, boost::asio::io_context&)+0x214) [0x5555b3b27724] 2023-10-18T16:09:03.436 INFO:tasks.ceph.mds.b.smithi151.stderr: 12: (MDSRankDispatcher::MDSRankDispatcher(int, std::__cxx11::basic_string, std::allocator >, ceph::fair_mutex&, std::shared_ptr&, CommonSafeTimer&, Beacon&, std::unique_ptr >&, Messenger*, MonClient*, MgrClient*, Context*, Context*, boost::asio::io_context&)+0x101) [0x5555b3b29181] 2023-10-18T16:09:03.436 INFO:tasks.ceph.mds.b.smithi151.stderr: 13: (MDSDaemon::handle_mds_map(boost::intrusive_ptr const&)+0x1608) [0x5555b3af9348] 2023-10-18T16:09:03.436 INFO:tasks.ceph.mds.b.smithi151.stderr: 14: (MDSDaemon::handle_core_message(boost::intrusive_ptr const&)+0xcd) [0x5555b3afc44d] 2023-10-18T16:09:03.436 INFO:tasks.ceph.mds.b.smithi151.stderr: 15: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0xc3) [0x5555b3afcfa3] 2023-10-18T16:09:03.436 INFO:tasks.ceph.mds.b.smithi151.stderr: 16: (DispatchQueue::entry()+0x126a) [0x7f225ca0d4ea] 2023-10-18T16:09:03.436 INFO:tasks.ceph.mds.b.smithi151.stderr: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f225cac1d51] 2023-10-18T16:09:03.437 INFO:tasks.ceph.mds.b.smithi151.stderr: 18: /lib64/libpthread.so.0(+0x814a) [0x7f225b7a914a] 2023-10-18T16:09:03.437 INFO:tasks.ceph.mds.b.smithi151.stderr: 19: clone() 2023-10-18T16:09:03.437 INFO:tasks.ceph.mds.b.smithi151.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-10-18T16:09:03.437 INFO:tasks.ceph.mds.b.smithi151.stderr: 2023-10-18T16:09:03.438 INFO:tasks.ceph.mds.b.smithi151.stderr: -522> 2023-10-18T16:07:36.646+0000 7f22577b7700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2023-10-18T16:09:03.438 INFO:tasks.ceph.mds.b.smithi151.stderr: 0> 2023-10-18T16:09:03.426+0000 7f2253fb0700 -1 *** Caught signal (Aborted) ** 2023-10-18T16:09:03.438 INFO:tasks.ceph.mds.b.smithi151.stderr: in thread 7f2253fb0700 thread_name:ms_dispatch 2023-10-18T16:09:03.438 INFO:tasks.ceph.mds.b.smithi151.stderr: 2023-10-18T16:09:03.439 INFO:tasks.ceph.mds.b.smithi151.stderr: ceph version 16.2.14-100-gb4ddd52c (b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5) pacific (stable) 2023-10-18T16:09:03.439 INFO:tasks.ceph.mds.b.smithi151.stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f225b7b3b20] 2023-10-18T16:09:03.439 INFO:tasks.ceph.mds.b.smithi151.stderr: 2: gsignal() 2023-10-18T16:09:03.439 INFO:tasks.ceph.mds.b.smithi151.stderr: 3: abort() 2023-10-18T16:09:03.439 INFO:tasks.ceph.mds.b.smithi151.stderr: 4: /lib64/libstdc++.so.6(+0x9009b) [0x7f225abb409b] 2023-10-18T16:09:03.439 INFO:tasks.ceph.mds.b.smithi151.stderr: 5: /lib64/libstdc++.so.6(+0x9653c) [0x7f225abba53c] 2023-10-18T16:09:03.440 INFO:tasks.ceph.mds.b.smithi151.stderr: 6: /lib64/libstdc++.so.6(+0x96597) [0x7f225abba597] 2023-10-18T16:09:03.440 INFO:tasks.ceph.mds.b.smithi151.stderr: 7: /lib64/libstdc++.so.6(+0x967f8) [0x7f225abba7f8] 2023-10-18T16:09:03.440 INFO:tasks.ceph.mds.b.smithi151.stderr: 8: ceph-mds(+0x212b43) [0x5555b3b4db43] 2023-10-18T16:09:03.440 INFO:tasks.ceph.mds.b.smithi151.stderr: 9: (Option::size_t const md_config_t::get_val(ConfigValues const&, std::basic_string_view >) const+0x5c) [0x5555b3bc289c] 2023-10-18T16:09:03.440 INFO:tasks.ceph.mds.b.smithi151.stderr: 10: (SessionMap::SessionMap(MDSRank*)+0x2c2) [0x5555b3e0a6a2] 2023-10-18T16:09:03.441 INFO:tasks.ceph.mds.b.smithi151.stderr: 11: (MDSRank::MDSRank(int, std::__cxx11::basic_string, std::allocator >, ceph::fair_mutex&, std::shared_ptr&, CommonSafeTimer&, Beacon&, std::unique_ptr >&, Messenger*, MonClient*, MgrClient*, Context*, Context*, boost::asio::io_context&)+0x214) [0x5555b3b27724] 2023-10-18T16:09:03.441 INFO:tasks.ceph.mds.b.smithi151.stderr: 12: (MDSRankDispatcher::MDSRankDispatcher(int, std::__cxx11::basic_string, std::allocator >, ceph::fair_mutex&, std::shared_ptr&, CommonSafeTimer&, Beacon&, std::unique_ptr >&, Messenger*, MonClient*, MgrClient*, Context*, Context*, boost::asio::io_context&)+0x101) [0x5555b3b29181] 2023-10-18T16:09:03.441 INFO:tasks.ceph.mds.b.smithi151.stderr: 13: (MDSDaemon::handle_mds_map(boost::intrusive_ptr const&)+0x1608) [0x5555b3af9348] 2023-10-18T16:09:03.441 INFO:tasks.ceph.mds.b.smithi151.stderr: 14: (MDSDaemon::handle_core_message(boost::intrusive_ptr const&)+0xcd) [0x5555b3afc44d] 2023-10-18T16:09:03.441 INFO:tasks.ceph.mds.b.smithi151.stderr: 15: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0xc3) [0x5555b3afcfa3] 2023-10-18T16:09:03.441 INFO:tasks.ceph.mds.b.smithi151.stderr: 16: (DispatchQueue::entry()+0x126a) [0x7f225ca0d4ea] 2023-10-18T16:09:03.442 INFO:tasks.ceph.mds.b.smithi151.stderr: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f225cac1d51] 2023-10-18T16:09:03.442 INFO:tasks.ceph.mds.b.smithi151.stderr: 18: /lib64/libpthread.so.0(+0x814a) [0x7f225b7a914a] 2023-10-18T16:09:03.442 INFO:tasks.ceph.mds.b.smithi151.stderr: 19: clone() 2023-10-18T16:09:03.442 INFO:tasks.ceph.mds.b.smithi151.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-10-18T16:09:03.442 INFO:tasks.ceph.mds.b.smithi151.stderr: 2023-10-18T16:09:03.443 INFO:tasks.ceph.mds.b.smithi151.stderr: -522> 2023-10-18T16:07:36.646+0000 7f22577b7700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2023-10-18T16:09:03.443 INFO:tasks.ceph.mds.b.smithi151.stderr: 0> 2023-10-18T16:09:03.426+0000 7f2253fb0700 -1 *** Caught signal (Aborted) ** 2023-10-18T16:09:03.443 INFO:tasks.ceph.mds.b.smithi151.stderr: in thread 7f2253fb0700 thread_name:ms_dispatch 2023-10-18T16:09:03.443 INFO:tasks.ceph.mds.b.smithi151.stderr: 2023-10-18T16:09:03.443 INFO:tasks.ceph.mds.b.smithi151.stderr: ceph version 16.2.14-100-gb4ddd52c (b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5) pacific (stable) 2023-10-18T16:09:03.443 INFO:tasks.ceph.mds.b.smithi151.stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f225b7b3b20] 2023-10-18T16:09:03.444 INFO:tasks.ceph.mds.b.smithi151.stderr: 2: gsignal() 2023-10-18T16:09:03.444 INFO:tasks.ceph.mds.b.smithi151.stderr: 3: abort() 2023-10-18T16:09:03.444 INFO:tasks.ceph.mds.b.smithi151.stderr: 4: /lib64/libstdc++.so.6(+0x9009b) [0x7f225abb409b] 2023-10-18T16:09:03.444 INFO:tasks.ceph.mds.b.smithi151.stderr: 5: /lib64/libstdc++.so.6(+0x9653c) [0x7f225abba53c] 2023-10-18T16:09:03.444 INFO:tasks.ceph.mds.b.smithi151.stderr: 6: /lib64/libstdc++.so.6(+0x96597) [0x7f225abba597] 2023-10-18T16:09:03.445 INFO:tasks.ceph.mds.b.smithi151.stderr: 7: /lib64/libstdc++.so.6(+0x967f8) [0x7f225abba7f8] 2023-10-18T16:09:03.445 INFO:tasks.ceph.mds.b.smithi151.stderr: 8: ceph-mds(+0x212b43) [0x5555b3b4db43] 2023-10-18T16:09:03.445 INFO:tasks.ceph.mds.b.smithi151.stderr: 9: (Option::size_t const md_config_t::get_val(ConfigValues const&, std::basic_string_view >) const+0x5c) [0x5555b3bc289c] 2023-10-18T16:09:03.445 INFO:tasks.ceph.mds.b.smithi151.stderr: 10: (SessionMap::SessionMap(MDSRank*)+0x2c2) [0x5555b3e0a6a2] 2023-10-18T16:09:03.445 INFO:tasks.ceph.mds.b.smithi151.stderr: 11: (MDSRank::MDSRank(int, std::__cxx11::basic_string, std::allocator >, ceph::fair_mutex&, std::shared_ptr&, CommonSafeTimer&, Beacon&, std::unique_ptr >&, Messenger*, MonClient*, MgrClient*, Context*, Context*, boost::asio::io_context&)+0x214) [0x5555b3b27724] 2023-10-18T16:09:03.446 INFO:tasks.ceph.mds.b.smithi151.stderr: 12: (MDSRankDispatcher::MDSRankDispatcher(int, std::__cxx11::basic_string, std::allocator >, ceph::fair_mutex&, std::shared_ptr&, CommonSafeTimer&, Beacon&, std::unique_ptr >&, Messenger*, MonClient*, MgrClient*, Context*, Context*, boost::asio::io_context&)+0x101) [0x5555b3b29181] 2023-10-18T16:09:03.446 INFO:tasks.ceph.mds.b.smithi151.stderr: 13: (MDSDaemon::handle_mds_map(boost::intrusive_ptr const&)+0x1608) [0x5555b3af9348] 2023-10-18T16:09:03.446 INFO:tasks.ceph.mds.b.smithi151.stderr: 14: (MDSDaemon::handle_core_message(boost::intrusive_ptr const&)+0xcd) [0x5555b3afc44d] 2023-10-18T16:09:03.446 INFO:tasks.ceph.mds.b.smithi151.stderr: 15: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0xc3) [0x5555b3afcfa3] 2023-10-18T16:09:03.446 INFO:tasks.ceph.mds.b.smithi151.stderr: 16: (DispatchQueue::entry()+0x126a) [0x7f225ca0d4ea] 2023-10-18T16:09:03.446 INFO:tasks.ceph.mds.b.smithi151.stderr: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f225cac1d51] 2023-10-18T16:09:03.447 INFO:tasks.ceph.mds.b.smithi151.stderr: 18: /lib64/libpthread.so.0(+0x814a) [0x7f225b7a914a] 2023-10-18T16:09:03.447 INFO:tasks.ceph.mds.b.smithi151.stderr: 19: clone() 2023-10-18T16:09:03.447 INFO:tasks.ceph.mds.b.smithi151.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-10-18T16:09:03.447 INFO:tasks.ceph.mds.b.smithi151.stderr: 2023-10-18T16:09:03.551 INFO:tasks.ceph.mds.b.smithi151.stderr:daemon-helper: command crashed with signal 6 2023-10-18T16:09:03.719 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~13s 2023-10-18T16:09:03.719 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~74s 2023-10-18T16:09:03.720 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.f is failed for ~27s 2023-10-18T16:09:04.763 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:09:04.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.884+0000 7f614b522700 1 Processor -- start 2023-10-18T16:09:04.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.884+0000 7f614b522700 1 -- start start 2023-10-18T16:09:04.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.885+0000 7f614b522700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f61440c3020 0x7f6144150ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:04.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.885+0000 7f614b522700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6144069f00 con 0x7f614414dc70 2023-10-18T16:09:04.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.885+0000 7f614b522700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f614406a1e0 con 0x7f614414d890 2023-10-18T16:09:04.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.885+0000 7f614b522700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61441513c0 con 0x7f61440c3020 2023-10-18T16:09:04.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.885+0000 7f6149abf700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f614414dc70 0x7f6144064920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:33560/0 (socket says 172.21.15.88:33560) 2023-10-18T16:09:04.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.885+0000 7f6149abf700 1 -- 172.21.15.88:0/3545647920 learned_addr learned my addr 172.21.15.88:0/3545647920 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:09:04.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.886+0000 7f6148abd700 1 --2- 172.21.15.88:0/3545647920 >> v2:172.21.15.151:3300/0 conn(0x7f61440c3020 0x7f6144150ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:04.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.886+0000 7f613bfff700 1 -- 172.21.15.88:0/3545647920 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4221796956 0 0) 0x7f6144069f00 con 0x7f614414dc70 2023-10-18T16:09:04.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.886+0000 7f613bfff700 1 -- 172.21.15.88:0/3545647920 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61300033f0 con 0x7f614414dc70 2023-10-18T16:09:04.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.886+0000 7f613bfff700 1 -- 172.21.15.88:0/3545647920 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2855744487 0 0) 0x7f61300033f0 con 0x7f614414dc70 2023-10-18T16:09:04.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.886+0000 7f613bfff700 1 -- 172.21.15.88:0/3545647920 >> v2:172.21.15.151:3300/0 conn(0x7f61440c3020 msgr2=0x7f6144150ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:09:04.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.886+0000 7f613bfff700 1 --2- 172.21.15.88:0/3545647920 >> v2:172.21.15.151:3300/0 conn(0x7f61440c3020 0x7f6144150ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:04.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.886+0000 7f613bfff700 1 -- 172.21.15.88:0/3545647920 >> v1:172.21.15.151:6789/0 conn(0x7f614414d890 legacy=0x7f61440a8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:04.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.886+0000 7f613bfff700 1 -- 172.21.15.88:0/3545647920 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f614406a4c0 con 0x7f614414dc70 2023-10-18T16:09:04.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.886+0000 7f613bfff700 1 -- 172.21.15.88:0/3545647920 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441715075 0 0) 0x7f613c0017f0 con 0x7f614414dc70 2023-10-18T16:09:04.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.886+0000 7f6148abd700 1 --2- 172.21.15.88:0/3545647920 >> v2:172.21.15.151:3300/0 conn(0x7f61440c3020 0x7f6144150ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2023-10-18T16:09:04.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.886+0000 7f613bfff700 1 -- 172.21.15.88:0/3545647920 >> v1:172.21.15.88:6789/0 conn(0x7f614414dc70 legacy=0x7f6144064920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:04.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.886+0000 7f613bfff700 1 --2- 172.21.15.88:0/3545647920 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f61300038e0 0x7f6130003d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:04.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.886+0000 7f613bfff700 1 --2- 172.21.15.88:0/3545647920 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6130004690 0x7f6130004b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:04.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.886+0000 7f613bfff700 1 --2- 172.21.15.88:0/3545647920 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6130005000 0x7f613000b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:04.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.886+0000 7f613bfff700 1 -- 172.21.15.88:0/3545647920 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61441513c0 con 0x7f6130004690 2023-10-18T16:09:04.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.886+0000 7f613bfff700 1 -- 172.21.15.88:0/3545647920 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f613000beb0 con 0x7f61300038e0 2023-10-18T16:09:04.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.886+0000 7f613bfff700 1 -- 172.21.15.88:0/3545647920 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f613000c070 con 0x7f6130005000 2023-10-18T16:09:04.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.886+0000 7f6148abd700 1 --2- 172.21.15.88:0/3545647920 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6130004690 0x7f6130004b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:04.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.886+0000 7f61492be700 1 --2- 172.21.15.88:0/3545647920 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f61300038e0 0x7f6130003d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:04.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.886+0000 7f6149abf700 1 --2- 172.21.15.88:0/3545647920 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6130005000 0x7f613000b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:04.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.886+0000 7f6148abd700 1 -- 172.21.15.88:0/3545647920 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6130005000 msgr2=0x7f613000b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:09:04.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.886+0000 7f6148abd700 1 --2- 172.21.15.88:0/3545647920 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6130005000 0x7f613000b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:04.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.886+0000 7f6148abd700 1 -- 172.21.15.88:0/3545647920 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f61300038e0 msgr2=0x7f6130003d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:09:04.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.886+0000 7f6148abd700 1 --2- 172.21.15.88:0/3545647920 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f61300038e0 0x7f6130003d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:04.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.886+0000 7f6148abd700 1 -- 172.21.15.88:0/3545647920 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61300033f0 con 0x7f6130004690 2023-10-18T16:09:04.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.886+0000 7f6149abf700 1 --2- 172.21.15.88:0/3545647920 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6130005000 0x7f613000b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-10-18T16:09:04.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.886+0000 7f6149abf700 1 -- 172.21.15.88:0/3545647920 reap_dead start 2023-10-18T16:09:04.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.886+0000 7f6148abd700 1 --2- 172.21.15.88:0/3545647920 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6130004690 0x7f6130004b10 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f6134009c70 tx=0x7f6134009060).ready entity=mon.0 client_cookie=8e6ab54ec5a1f1d1 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:09:04.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.887+0000 7f613bfff700 1 -- 172.21.15.88:0/3545647920 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61340095c0 con 0x7f6130004690 2023-10-18T16:09:04.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.887+0000 7f613bfff700 1 -- 172.21.15.88:0/3545647920 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f613400e6f0 con 0x7f6130004690 2023-10-18T16:09:04.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.887+0000 7f613bfff700 1 -- 172.21.15.88:0/3545647920 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f613400ea70 con 0x7f6130004690 2023-10-18T16:09:04.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.887+0000 7f614b522700 1 -- 172.21.15.88:0/3545647920 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6130004690 msgr2=0x7f6130004b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:04.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.887+0000 7f614b522700 1 --2- 172.21.15.88:0/3545647920 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6130004690 0x7f6130004b10 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f6134009c70 tx=0x7f6134009060).stop 2023-10-18T16:09:04.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.887+0000 7f614b522700 1 -- 172.21.15.88:0/3545647920 shutdown_connections 2023-10-18T16:09:04.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.887+0000 7f614b522700 1 --2- 172.21.15.88:0/3545647920 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6130004690 0x7f6130004b10 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:04.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.887+0000 7f614b522700 1 -- 172.21.15.88:0/3545647920 >> 172.21.15.88:0/3545647920 conn(0x7f61440c6a50 msgr2=0x7f61440c6e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:09:04.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.887+0000 7f614b522700 1 -- 172.21.15.88:0/3545647920 shutdown_connections 2023-10-18T16:09:04.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.887+0000 7f614b522700 1 -- 172.21.15.88:0/3545647920 wait complete. 2023-10-18T16:09:04.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.888+0000 7f614b522700 1 Processor -- start 2023-10-18T16:09:04.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.888+0000 7f614b522700 1 -- start start 2023-10-18T16:09:04.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.889+0000 7f614b522700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6130004690 0x7f6144151c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:04.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.889+0000 7f614b522700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f61440ac760 0x7f61440bea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:04.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.889+0000 7f614b522700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f61440bfa80 0x7f61440abcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:04.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.889+0000 7f614b522700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61440bff00 con 0x7f6130004690 2023-10-18T16:09:04.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.889+0000 7f614b522700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6144160fa0 con 0x7f61440bfa80 2023-10-18T16:09:04.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.889+0000 7f614b522700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6144160a60 con 0x7f61440ac760 2023-10-18T16:09:04.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.889+0000 7f61492be700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6130004690 0x7f6144151c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:04.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.889+0000 7f61492be700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6130004690 0x7f6144151c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35710/0 (socket says 172.21.15.88:35710) 2023-10-18T16:09:04.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.889+0000 7f61492be700 1 -- 172.21.15.88:0/3586492805 learned_addr learned my addr 172.21.15.88:0/3586492805 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:09:04.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.890+0000 7f6148abd700 1 --2- 172.21.15.88:0/3586492805 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f61440ac760 0x7f61440bea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:04.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.890+0000 7f61492be700 1 -- 172.21.15.88:0/3586492805 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f61440ac760 msgr2=0x7f61440bea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:04.905 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.890+0000 7f61492be700 1 --2- 172.21.15.88:0/3586492805 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f61440ac760 0x7f61440bea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:04.905 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.890+0000 7f6149abf700 1 --2- 172.21.15.88:0/3586492805 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f61440bfa80 0x7f61440abcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:04.905 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.890+0000 7f61492be700 1 -- 172.21.15.88:0/3586492805 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f61440bfa80 msgr2=0x7f61440abcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:04.905 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.890+0000 7f61492be700 1 --2- 172.21.15.88:0/3586492805 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f61440bfa80 0x7f61440abcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:04.905 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.890+0000 7f61492be700 1 -- 172.21.15.88:0/3586492805 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61440bdf60 con 0x7f6130004690 2023-10-18T16:09:04.906 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.890+0000 7f61492be700 1 --2- 172.21.15.88:0/3586492805 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6130004690 0x7f6144151c40 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f61400028f0 tx=0x7f6140007bb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:09:04.906 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.890+0000 7f61397fa700 1 -- 172.21.15.88:0/3586492805 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f614000f6c0 con 0x7f6130004690 2023-10-18T16:09:04.906 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.890+0000 7f61397fa700 1 -- 172.21.15.88:0/3586492805 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f614000f820 con 0x7f6130004690 2023-10-18T16:09:04.906 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.890+0000 7f614b522700 1 -- 172.21.15.88:0/3586492805 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6144160cc0 con 0x7f6130004690 2023-10-18T16:09:04.906 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.890+0000 7f61397fa700 1 -- 172.21.15.88:0/3586492805 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f614000fba0 con 0x7f6130004690 2023-10-18T16:09:04.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.890+0000 7f614b522700 1 -- 172.21.15.88:0/3586492805 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61440be4e0 con 0x7f6130004690 2023-10-18T16:09:04.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.891+0000 7f614b522700 1 -- 172.21.15.88:0/3586492805 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61440bddc0 con 0x7f6130004690 2023-10-18T16:09:04.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.891+0000 7f61397fa700 1 -- 172.21.15.88:0/3586492805 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f6140014450 con 0x7f6130004690 2023-10-18T16:09:04.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.892+0000 7f61397fa700 1 --2- 172.21.15.88:0/3586492805 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f612805f150 0x7f6128061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:04.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.892+0000 7f61397fa700 1 -- 172.21.15.88:0/3586492805 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6553+0+0 (secure 0 0 0) 0x7f614007a060 con 0x7f6130004690 2023-10-18T16:09:04.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.892+0000 7f6148abd700 1 --2- 172.21.15.88:0/3586492805 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f612805f150 0x7f6128061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:04.908 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.892+0000 7f6148abd700 1 --2- 172.21.15.88:0/3586492805 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f612805f150 0x7f6128061610 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f61440acba0 tx=0x7f613401a000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:09:04.908 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:04.896+0000 7f61397fa700 1 -- 172.21.15.88:0/3586492805 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f61440bddc0 con 0x7f6130004690 2023-10-18T16:09:05.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:05.094+0000 7f614b522700 1 -- 172.21.15.88:0/3586492805 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f61440bddc0 con 0x7f6130004690 2023-10-18T16:09:05.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:05.095+0000 7f61397fa700 1 -- 172.21.15.88:0/3586492805 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7f61440bddc0 con 0x7f6130004690 2023-10-18T16:09:05.099 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:09:05.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:05.097+0000 7f614b522700 1 -- 172.21.15.88:0/3586492805 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f612805f150 msgr2=0x7f6128061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:05.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:05.097+0000 7f614b522700 1 --2- 172.21.15.88:0/3586492805 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f612805f150 0x7f6128061610 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f61440acba0 tx=0x7f613401a000).stop 2023-10-18T16:09:05.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:05.097+0000 7f614b522700 1 -- 172.21.15.88:0/3586492805 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6130004690 msgr2=0x7f6144151c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:05.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:05.097+0000 7f614b522700 1 --2- 172.21.15.88:0/3586492805 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6130004690 0x7f6144151c40 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f61400028f0 tx=0x7f6140007bb0).stop 2023-10-18T16:09:05.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:05.097+0000 7f614b522700 1 -- 172.21.15.88:0/3586492805 shutdown_connections 2023-10-18T16:09:05.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:05.097+0000 7f614b522700 1 --2- 172.21.15.88:0/3586492805 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f612805f150 0x7f6128061610 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:05.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:05.097+0000 7f614b522700 1 --2- 172.21.15.88:0/3586492805 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f61440bfa80 0x7f61440abcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:05.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:05.097+0000 7f614b522700 1 --2- 172.21.15.88:0/3586492805 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6130004690 0x7f6144151c40 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:05.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:05.097+0000 7f614b522700 1 --2- 172.21.15.88:0/3586492805 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f61440ac760 0x7f61440bea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:05.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:05.097+0000 7f614b522700 1 -- 172.21.15.88:0/3586492805 >> 172.21.15.88:0/3586492805 conn(0x7f61440c6a50 msgr2=0x7f61440c2610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:09:05.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:05.097+0000 7f614b522700 1 -- 172.21.15.88:0/3586492805 shutdown_connections 2023-10-18T16:09:05.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:05.097+0000 7f614b522700 1 -- 172.21.15.88:0/3586492805 wait complete. 2023-10-18T16:09:05.112 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:09:08.113 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:09:08.238 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.234+0000 7f76a1268700 1 Processor -- start 2023-10-18T16:09:08.238 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.234+0000 7f76a1268700 1 -- start start 2023-10-18T16:09:08.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.235+0000 7f76a1268700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f769c14d8c0 0x7f769c064440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:08.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.235+0000 7f76a1268700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f769c065360 0x7f769c064930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:08.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.235+0000 7f76a1268700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f769c064150 con 0x7f769c0a9050 2023-10-18T16:09:08.241 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.235+0000 7f76a1268700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f769c153a40 con 0x7f769c065360 2023-10-18T16:09:08.241 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.235+0000 7f76a1268700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f769c063e70 con 0x7f769c14d8c0 2023-10-18T16:09:08.241 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.235+0000 7f769b7fe700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f769c065360 0x7f769c064930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:08.241 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.235+0000 7f769b7fe700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f769c065360 0x7f769c064930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56084/0 (socket says 172.21.15.88:56084) 2023-10-18T16:09:08.242 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.235+0000 7f769b7fe700 1 -- 172.21.15.88:0/101734201 learned_addr learned my addr 172.21.15.88:0/101734201 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:09:08.242 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.235+0000 7f769a7fc700 1 --1- 172.21.15.88:0/101734201 >> v1:172.21.15.88:6789/0 conn(0x7f769c0a9050 0x7f769c153ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:52082/0 (socket says 172.21.15.88:52082) 2023-10-18T16:09:08.242 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.235+0000 7f769affd700 1 --2- 172.21.15.88:0/101734201 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f769c14d8c0 0x7f769c064440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:08.242 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.235+0000 7f769b7fe700 1 -- 172.21.15.88:0/101734201 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f769c14d8c0 msgr2=0x7f769c064440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:09:08.242 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.235+0000 7f769b7fe700 1 --2- 172.21.15.88:0/101734201 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f769c14d8c0 0x7f769c064440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:08.243 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.235+0000 7f769b7fe700 1 -- 172.21.15.88:0/101734201 >> v1:172.21.15.88:6789/0 conn(0x7f769c0a9050 legacy=0x7f769c153ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:08.243 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.235+0000 7f7699ffb700 1 -- 172.21.15.88:0/101734201 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1255968118 0 0) 0x7f769c064150 con 0x7f769c0a9050 2023-10-18T16:09:08.243 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.235+0000 7f769b7fe700 1 -- 172.21.15.88:0/101734201 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f769c0c72c0 con 0x7f769c065360 2023-10-18T16:09:08.243 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.235+0000 7f769b7fe700 1 --2- 172.21.15.88:0/101734201 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f769c065360 0x7f769c064930 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7f768c006fd0 tx=0x7f768c0090f0).ready entity=mon.0 client_cookie=cfe83b8a6f9c7b49 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:09:08.243 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.235+0000 7f7699ffb700 1 -- 172.21.15.88:0/101734201 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f768c00b600 con 0x7f769c065360 2023-10-18T16:09:08.243 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.235+0000 7f7699ffb700 1 -- 172.21.15.88:0/101734201 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f768c00b760 con 0x7f769c065360 2023-10-18T16:09:08.244 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.235+0000 7f7699ffb700 1 -- 172.21.15.88:0/101734201 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f768c00bb00 con 0x7f769c065360 2023-10-18T16:09:08.244 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.236+0000 7f76a1268700 1 -- 172.21.15.88:0/101734201 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f769c065360 msgr2=0x7f769c064930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:08.244 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.236+0000 7f76a1268700 1 --2- 172.21.15.88:0/101734201 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f769c065360 0x7f769c064930 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7f768c006fd0 tx=0x7f768c0090f0).stop 2023-10-18T16:09:08.244 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.236+0000 7f76a1268700 1 -- 172.21.15.88:0/101734201 shutdown_connections 2023-10-18T16:09:08.244 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.236+0000 7f76a1268700 1 --2- 172.21.15.88:0/101734201 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f769c065360 0x7f769c064930 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:08.244 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.236+0000 7f76a1268700 1 --2- 172.21.15.88:0/101734201 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f769c14d8c0 0x7f769c064440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:08.245 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.236+0000 7f76a1268700 1 -- 172.21.15.88:0/101734201 >> 172.21.15.88:0/101734201 conn(0x7f769c06a810 msgr2=0x7f769c0c6a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:09:08.245 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.236+0000 7f76a1268700 1 -- 172.21.15.88:0/101734201 shutdown_connections 2023-10-18T16:09:08.245 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.236+0000 7f76a1268700 1 -- 172.21.15.88:0/101734201 wait complete. 2023-10-18T16:09:08.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.237+0000 7f76a1268700 1 Processor -- start 2023-10-18T16:09:08.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.237+0000 7f76a1268700 1 -- start start 2023-10-18T16:09:08.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.238+0000 7f76a1268700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f769c065360 0x7f769c0c60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:08.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.238+0000 7f76a1268700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f769c0a9050 0x7f769c1635c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:08.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.238+0000 7f76a1268700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f769c14d8c0 0x7f769c163ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:08.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.238+0000 7f76a1268700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f769c0bb360 con 0x7f769c065360 2023-10-18T16:09:08.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.238+0000 7f76a1268700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f769c152c50 con 0x7f769c14d8c0 2023-10-18T16:09:08.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.238+0000 7f76a1268700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f769c0ac050 con 0x7f769c0a9050 2023-10-18T16:09:08.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.238+0000 7f769affd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f769c065360 0x7f769c0c60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:08.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.238+0000 7f769affd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f769c065360 0x7f769c0c60a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56098/0 (socket says 172.21.15.88:56098) 2023-10-18T16:09:08.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.238+0000 7f769affd700 1 -- 172.21.15.88:0/410657595 learned_addr learned my addr 172.21.15.88:0/410657595 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:09:08.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.238+0000 7f769a7fc700 1 --2- 172.21.15.88:0/410657595 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f769c0a9050 0x7f769c1635c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:08.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.238+0000 7f769b7fe700 1 --2- 172.21.15.88:0/410657595 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f769c14d8c0 0x7f769c163ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:08.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.238+0000 7f769affd700 1 -- 172.21.15.88:0/410657595 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f769c0a9050 msgr2=0x7f769c1635c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:08.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.238+0000 7f769affd700 1 --2- 172.21.15.88:0/410657595 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f769c0a9050 0x7f769c1635c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:08.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.238+0000 7f769affd700 1 -- 172.21.15.88:0/410657595 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f769c14d8c0 msgr2=0x7f769c163ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:08.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.238+0000 7f769affd700 1 --2- 172.21.15.88:0/410657595 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f769c14d8c0 0x7f769c163ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:08.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.238+0000 7f769affd700 1 -- 172.21.15.88:0/410657595 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f768c009370 con 0x7f769c065360 2023-10-18T16:09:08.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.238+0000 7f769affd700 1 --2- 172.21.15.88:0/410657595 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f769c065360 0x7f769c0c60a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f769400ce40 tx=0x7f7694007810).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:09:08.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.239+0000 7f7692ffd700 1 -- 172.21.15.88:0/410657595 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f769400ab50 con 0x7f769c065360 2023-10-18T16:09:08.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.239+0000 7f7692ffd700 1 -- 172.21.15.88:0/410657595 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f769400acb0 con 0x7f769c065360 2023-10-18T16:09:08.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.239+0000 7f76a1268700 1 -- 172.21.15.88:0/410657595 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f769c0baa30 con 0x7f769c065360 2023-10-18T16:09:08.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.239+0000 7f7692ffd700 1 -- 172.21.15.88:0/410657595 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f769400f610 con 0x7f769c065360 2023-10-18T16:09:08.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.239+0000 7f76a1268700 1 -- 172.21.15.88:0/410657595 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f769c0baff0 con 0x7f769c065360 2023-10-18T16:09:08.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.240+0000 7f76a1268700 1 -- 172.21.15.88:0/410657595 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f769c06c4e0 con 0x7f769c065360 2023-10-18T16:09:08.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.240+0000 7f7692ffd700 1 -- 172.21.15.88:0/410657595 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f769400d040 con 0x7f769c065360 2023-10-18T16:09:08.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.240+0000 7f7692ffd700 1 --2- 172.21.15.88:0/410657595 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f767c05f120 0x7f767c0615e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:08.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.240+0000 7f7692ffd700 1 -- 172.21.15.88:0/410657595 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6553+0+0 (secure 0 0 0) 0x7f769407a590 con 0x7f769c065360 2023-10-18T16:09:08.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.240+0000 7f769a7fc700 1 --2- 172.21.15.88:0/410657595 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f767c05f120 0x7f767c0615e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:08.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.241+0000 7f769a7fc700 1 --2- 172.21.15.88:0/410657595 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f767c05f120 0x7f767c0615e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f7684007fd0 tx=0x7f7684009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:09:08.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.244+0000 7f7692ffd700 1 -- 172.21.15.88:0/410657595 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f769404b6b0 con 0x7f769c065360 2023-10-18T16:09:08.364 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-18T16:09:08.366 DEBUG:teuthology.orchestra.run.smithi151:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-18T16:09:08.454 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.451+0000 7f76a1268700 1 -- 172.21.15.88:0/410657595 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f769c0bb6f0 con 0x7f769c065360 2023-10-18T16:09:08.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.451+0000 7f7692ffd700 1 -- 172.21.15.88:0/410657595 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7f769401d0e0 con 0x7f769c065360 2023-10-18T16:09:08.456 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:09:08.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.453+0000 7f76a1268700 1 -- 172.21.15.88:0/410657595 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f767c05f120 msgr2=0x7f767c0615e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:08.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.453+0000 7f76a1268700 1 --2- 172.21.15.88:0/410657595 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f767c05f120 0x7f767c0615e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f7684007fd0 tx=0x7f7684009040).stop 2023-10-18T16:09:08.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.453+0000 7f76a1268700 1 -- 172.21.15.88:0/410657595 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f769c065360 msgr2=0x7f769c0c60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:08.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.453+0000 7f76a1268700 1 --2- 172.21.15.88:0/410657595 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f769c065360 0x7f769c0c60a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f769400ce40 tx=0x7f7694007810).stop 2023-10-18T16:09:08.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.453+0000 7f76a1268700 1 -- 172.21.15.88:0/410657595 shutdown_connections 2023-10-18T16:09:08.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.453+0000 7f76a1268700 1 --2- 172.21.15.88:0/410657595 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f767c05f120 0x7f767c0615e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:08.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.453+0000 7f76a1268700 1 --2- 172.21.15.88:0/410657595 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f769c14d8c0 0x7f769c163ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:08.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.453+0000 7f76a1268700 1 --2- 172.21.15.88:0/410657595 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f769c065360 0x7f769c0c60a0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:08.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.453+0000 7f76a1268700 1 --2- 172.21.15.88:0/410657595 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f769c0a9050 0x7f769c1635c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:08.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.453+0000 7f76a1268700 1 -- 172.21.15.88:0/410657595 >> 172.21.15.88:0/410657595 conn(0x7f769c06a810 msgr2=0x7f769c151810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:09:08.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.453+0000 7f76a1268700 1 -- 172.21.15.88:0/410657595 shutdown_connections 2023-10-18T16:09:08.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:08.453+0000 7f76a1268700 1 -- 172.21.15.88:0/410657595 wait complete. 2023-10-18T16:09:08.468 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:09:10.231 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~20s 2023-10-18T16:09:10.232 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2023-10-18T16:09:10.232 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~81s 2023-10-18T16:09:10.232 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.f is failed for ~33s 2023-10-18T16:09:11.470 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:09:11.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.591+0000 7f1aa15c8700 1 Processor -- start 2023-10-18T16:09:11.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.591+0000 7f1aa15c8700 1 -- start start 2023-10-18T16:09:11.601 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.592+0000 7f1aa15c8700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f1a9c0a9050 0x7f1a9c153ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:11.601 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.592+0000 7f1aa15c8700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a9c064150 con 0x7f1a9c065360 2023-10-18T16:09:11.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.592+0000 7f1aa15c8700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a9c153a40 con 0x7f1a9c14d8c0 2023-10-18T16:09:11.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.592+0000 7f1aa15c8700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a9c063e70 con 0x7f1a9c0a9050 2023-10-18T16:09:11.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.592+0000 7f1a9b7fe700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f1a9c065360 0x7f1a9c15acb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:52092/0 (socket says 172.21.15.88:52092) 2023-10-18T16:09:11.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.592+0000 7f1a9b7fe700 1 -- 172.21.15.88:0/3939842129 learned_addr learned my addr 172.21.15.88:0/3939842129 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:09:11.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.593+0000 7f1a9a7fc700 1 --2- 172.21.15.88:0/3939842129 >> v2:172.21.15.151:3300/0 conn(0x7f1a9c0a9050 0x7f1a9c153ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:11.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.593+0000 7f1a99ffb700 1 -- 172.21.15.88:0/3939842129 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3793150508 0 0) 0x7f1a9c064150 con 0x7f1a9c065360 2023-10-18T16:09:11.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.593+0000 7f1a99ffb700 1 -- 172.21.15.88:0/3939842129 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a840033f0 con 0x7f1a9c065360 2023-10-18T16:09:11.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.593+0000 7f1a99ffb700 1 -- 172.21.15.88:0/3939842129 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1756225928 0 0) 0x7f1a840033f0 con 0x7f1a9c065360 2023-10-18T16:09:11.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.593+0000 7f1a99ffb700 1 -- 172.21.15.88:0/3939842129 >> v2:172.21.15.151:3300/0 conn(0x7f1a9c0a9050 msgr2=0x7f1a9c153ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:09:11.604 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.593+0000 7f1a99ffb700 1 --2- 172.21.15.88:0/3939842129 >> v2:172.21.15.151:3300/0 conn(0x7f1a9c0a9050 0x7f1a9c153ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:11.604 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.593+0000 7f1a99ffb700 1 -- 172.21.15.88:0/3939842129 >> v1:172.21.15.151:6789/0 conn(0x7f1a9c14d8c0 legacy=0x7f1a9c064440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:11.604 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.593+0000 7f1a99ffb700 1 -- 172.21.15.88:0/3939842129 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a9c0c72c0 con 0x7f1a9c065360 2023-10-18T16:09:11.604 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.593+0000 7f1a99ffb700 1 -- 172.21.15.88:0/3939842129 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441715075 0 0) 0x7f1a94001470 con 0x7f1a9c065360 2023-10-18T16:09:11.605 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.593+0000 7f1a99ffb700 1 -- 172.21.15.88:0/3939842129 >> v1:172.21.15.88:6789/0 conn(0x7f1a9c065360 legacy=0x7f1a9c15acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:11.605 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.593+0000 7f1a99ffb700 1 --2- 172.21.15.88:0/3939842129 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1a840038e0 0x7f1a84003d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:11.606 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.593+0000 7f1a99ffb700 1 --2- 172.21.15.88:0/3939842129 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a84004690 0x7f1a84004b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:11.606 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.593+0000 7f1a99ffb700 1 --2- 172.21.15.88:0/3939842129 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1a84005050 0x7f1a8400b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:11.606 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.593+0000 7f1a99ffb700 1 -- 172.21.15.88:0/3939842129 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a9c063e70 con 0x7f1a84004690 2023-10-18T16:09:11.606 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.593+0000 7f1a99ffb700 1 -- 172.21.15.88:0/3939842129 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a8400bfe0 con 0x7f1a84005050 2023-10-18T16:09:11.606 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.593+0000 7f1a99ffb700 1 -- 172.21.15.88:0/3939842129 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a8400c1a0 con 0x7f1a840038e0 2023-10-18T16:09:11.607 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.593+0000 7f1a9a7fc700 1 --2- 172.21.15.88:0/3939842129 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a84004690 0x7f1a84004b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:11.607 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.593+0000 7f1a9b7fe700 1 --2- 172.21.15.88:0/3939842129 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1a84005050 0x7f1a8400b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:11.607 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.593+0000 7f1a9affd700 1 --2- 172.21.15.88:0/3939842129 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1a840038e0 0x7f1a84003d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:11.607 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.593+0000 7f1a9a7fc700 1 -- 172.21.15.88:0/3939842129 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1a840038e0 msgr2=0x7f1a84003d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:09:11.607 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.593+0000 7f1a9a7fc700 1 --2- 172.21.15.88:0/3939842129 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1a840038e0 0x7f1a84003d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:11.608 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.593+0000 7f1a9a7fc700 1 -- 172.21.15.88:0/3939842129 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1a84005050 msgr2=0x7f1a8400b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:09:11.608 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.593+0000 7f1a9a7fc700 1 --2- 172.21.15.88:0/3939842129 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1a84005050 0x7f1a8400b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:11.608 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.593+0000 7f1a9a7fc700 1 -- 172.21.15.88:0/3939842129 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a840033f0 con 0x7f1a84004690 2023-10-18T16:09:11.608 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.593+0000 7f1a9b7fe700 1 --2- 172.21.15.88:0/3939842129 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1a84005050 0x7f1a8400b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:09:11.608 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.593+0000 7f1a9b7fe700 1 -- 172.21.15.88:0/3939842129 reap_dead start 2023-10-18T16:09:11.609 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.593+0000 7f1a9a7fc700 1 --2- 172.21.15.88:0/3939842129 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a84004690 0x7f1a84004b10 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f1a90006d80 tx=0x7f1a90009060).ready entity=mon.0 client_cookie=5eb8f9e6895ad989 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:09:11.609 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.594+0000 7f1a99ffb700 1 -- 172.21.15.88:0/3939842129 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a90009e90 con 0x7f1a84004690 2023-10-18T16:09:11.609 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.594+0000 7f1a99ffb700 1 -- 172.21.15.88:0/3939842129 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a9000e640 con 0x7f1a84004690 2023-10-18T16:09:11.609 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.594+0000 7f1a99ffb700 1 -- 172.21.15.88:0/3939842129 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a9000ea00 con 0x7f1a84004690 2023-10-18T16:09:11.609 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.594+0000 7f1aa15c8700 1 -- 172.21.15.88:0/3939842129 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a84004690 msgr2=0x7f1a84004b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:11.610 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.594+0000 7f1aa15c8700 1 --2- 172.21.15.88:0/3939842129 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a84004690 0x7f1a84004b10 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f1a90006d80 tx=0x7f1a90009060).stop 2023-10-18T16:09:11.610 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.594+0000 7f1aa15c8700 1 -- 172.21.15.88:0/3939842129 shutdown_connections 2023-10-18T16:09:11.610 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.594+0000 7f1aa15c8700 1 --2- 172.21.15.88:0/3939842129 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a84004690 0x7f1a84004b10 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:11.610 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.594+0000 7f1aa15c8700 1 -- 172.21.15.88:0/3939842129 >> 172.21.15.88:0/3939842129 conn(0x7f1a9c06a810 msgr2=0x7f1a9c0c6a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:09:11.610 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.594+0000 7f1aa15c8700 1 -- 172.21.15.88:0/3939842129 shutdown_connections 2023-10-18T16:09:11.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.594+0000 7f1aa15c8700 1 -- 172.21.15.88:0/3939842129 wait complete. 2023-10-18T16:09:11.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.595+0000 7f1aa15c8700 1 Processor -- start 2023-10-18T16:09:11.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.595+0000 7f1aa15c8700 1 -- start start 2023-10-18T16:09:11.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.596+0000 7f1aa15c8700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1a84004690 0x7f1a9c0a9a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:11.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.596+0000 7f1aa15c8700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1a9c160ce0 0x7f1a9c14e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:11.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.596+0000 7f1aa15c8700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a9c0bcba0 0x7f1a9c148c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:11.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.596+0000 7f1aa15c8700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a9c14f440 con 0x7f1a9c0bcba0 2023-10-18T16:09:11.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.596+0000 7f1aa15c8700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a9c14e390 con 0x7f1a84004690 2023-10-18T16:09:11.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.596+0000 7f1aa15c8700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a9c1506b0 con 0x7f1a9c160ce0 2023-10-18T16:09:11.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.596+0000 7f1a9b7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a9c0bcba0 0x7f1a9c148c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:11.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.596+0000 7f1a9b7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a9c0bcba0 0x7f1a9c148c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56122/0 (socket says 172.21.15.88:56122) 2023-10-18T16:09:11.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.596+0000 7f1a9b7fe700 1 -- 172.21.15.88:0/3251724861 learned_addr learned my addr 172.21.15.88:0/3251724861 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:09:11.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.596+0000 7f1a9affd700 1 --2- 172.21.15.88:0/3251724861 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1a84004690 0x7f1a9c0a9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:11.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.596+0000 7f1a9a7fc700 1 --2- 172.21.15.88:0/3251724861 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1a9c160ce0 0x7f1a9c14e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:11.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.596+0000 7f1a9b7fe700 1 -- 172.21.15.88:0/3251724861 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1a9c160ce0 msgr2=0x7f1a9c14e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:11.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.596+0000 7f1a9b7fe700 1 --2- 172.21.15.88:0/3251724861 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1a9c160ce0 0x7f1a9c14e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:11.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.596+0000 7f1a9b7fe700 1 -- 172.21.15.88:0/3251724861 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1a84004690 msgr2=0x7f1a9c0a9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:11.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.596+0000 7f1a9b7fe700 1 --2- 172.21.15.88:0/3251724861 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1a84004690 0x7f1a9c0a9a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:11.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.596+0000 7f1a9b7fe700 1 -- 172.21.15.88:0/3251724861 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a9c150c50 con 0x7f1a9c0bcba0 2023-10-18T16:09:11.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.597+0000 7f1a9b7fe700 1 --2- 172.21.15.88:0/3251724861 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a9c0bcba0 0x7f1a9c148c60 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f1a94000c00 tx=0x7f1a94000ed0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:09:11.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.597+0000 7f1a82ffd700 1 -- 172.21.15.88:0/3251724861 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a94010450 con 0x7f1a9c0bcba0 2023-10-18T16:09:11.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.597+0000 7f1a82ffd700 1 -- 172.21.15.88:0/3251724861 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a940105d0 con 0x7f1a9c0bcba0 2023-10-18T16:09:11.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.597+0000 7f1aa15c8700 1 -- 172.21.15.88:0/3251724861 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a9c161160 con 0x7f1a9c0bcba0 2023-10-18T16:09:11.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.597+0000 7f1a82ffd700 1 -- 172.21.15.88:0/3251724861 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a94010990 con 0x7f1a9c0bcba0 2023-10-18T16:09:11.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.597+0000 7f1aa15c8700 1 -- 172.21.15.88:0/3251724861 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a9c14dd40 con 0x7f1a9c0bcba0 2023-10-18T16:09:11.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.598+0000 7f1a82ffd700 1 -- 172.21.15.88:0/3251724861 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f1a94010d40 con 0x7f1a9c0bcba0 2023-10-18T16:09:11.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.598+0000 7f1aa15c8700 1 -- 172.21.15.88:0/3251724861 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a9c14e500 con 0x7f1a9c0bcba0 2023-10-18T16:09:11.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.598+0000 7f1a82ffd700 1 --2- 172.21.15.88:0/3251724861 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f1a8805f150 0x7f1a88061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:11.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.599+0000 7f1a82ffd700 1 -- 172.21.15.88:0/3251724861 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6553+0+0 (secure 0 0 0) 0x7f1a9407a0e0 con 0x7f1a9c0bcba0 2023-10-18T16:09:11.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.599+0000 7f1a9affd700 1 --2- 172.21.15.88:0/3251724861 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f1a8805f150 0x7f1a88061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:11.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.599+0000 7f1a9affd700 1 --2- 172.21.15.88:0/3251724861 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f1a8805f150 0x7f1a88061610 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f1a8c000f80 tx=0x7f1a8c009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:09:11.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.603+0000 7f1a82ffd700 1 -- 172.21.15.88:0/3251724861 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f1a9404b200 con 0x7f1a9c0bcba0 2023-10-18T16:09:11.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.804+0000 7f1aa15c8700 1 -- 172.21.15.88:0/3251724861 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f1a9c0c75a0 con 0x7f1a9c0bcba0 2023-10-18T16:09:11.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.804+0000 7f1a82ffd700 1 -- 172.21.15.88:0/3251724861 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7f1a9404e820 con 0x7f1a9c0bcba0 2023-10-18T16:09:11.809 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:09:11.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.806+0000 7f1aa15c8700 1 -- 172.21.15.88:0/3251724861 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f1a8805f150 msgr2=0x7f1a88061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:11.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.806+0000 7f1aa15c8700 1 --2- 172.21.15.88:0/3251724861 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f1a8805f150 0x7f1a88061610 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f1a8c000f80 tx=0x7f1a8c009040).stop 2023-10-18T16:09:11.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.806+0000 7f1aa15c8700 1 -- 172.21.15.88:0/3251724861 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a9c0bcba0 msgr2=0x7f1a9c148c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:11.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.806+0000 7f1aa15c8700 1 --2- 172.21.15.88:0/3251724861 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a9c0bcba0 0x7f1a9c148c60 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f1a94000c00 tx=0x7f1a94000ed0).stop 2023-10-18T16:09:11.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.806+0000 7f1aa15c8700 1 -- 172.21.15.88:0/3251724861 shutdown_connections 2023-10-18T16:09:11.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.806+0000 7f1aa15c8700 1 --2- 172.21.15.88:0/3251724861 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f1a8805f150 0x7f1a88061610 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:11.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.806+0000 7f1aa15c8700 1 --2- 172.21.15.88:0/3251724861 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a9c0bcba0 0x7f1a9c148c60 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:11.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.806+0000 7f1aa15c8700 1 --2- 172.21.15.88:0/3251724861 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1a84004690 0x7f1a9c0a9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:11.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.806+0000 7f1aa15c8700 1 --2- 172.21.15.88:0/3251724861 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1a9c160ce0 0x7f1a9c14e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:11.812 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.806+0000 7f1aa15c8700 1 -- 172.21.15.88:0/3251724861 >> 172.21.15.88:0/3251724861 conn(0x7f1a9c06a810 msgr2=0x7f1a9c153d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:09:11.812 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.806+0000 7f1aa15c8700 1 -- 172.21.15.88:0/3251724861 shutdown_connections 2023-10-18T16:09:11.812 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:11.806+0000 7f1aa15c8700 1 -- 172.21.15.88:0/3251724861 wait complete. 2023-10-18T16:09:11.821 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:09:14.822 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:09:14.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:14.945+0000 7f84e94d7700 1 Processor -- start 2023-10-18T16:09:14.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:14.945+0000 7f84e94d7700 1 -- start start 2023-10-18T16:09:14.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:14.946+0000 7f84e94d7700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f84e40c6a60 0x7f84e40c6e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:14.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:14.946+0000 7f84e94d7700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f84e4063d70 0x7f84e4153660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:14.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:14.946+0000 7f84e94d7700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84e4151020 con 0x7f84e4153b50 2023-10-18T16:09:14.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:14.946+0000 7f84e94d7700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84e4150090 con 0x7f84e4063d70 2023-10-18T16:09:14.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:14.946+0000 7f84e94d7700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84e4150b90 con 0x7f84e40c6a60 2023-10-18T16:09:14.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:14.946+0000 7f84e37fe700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f84e4063d70 0x7f84e4153660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:14.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:14.946+0000 7f84e37fe700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f84e4063d70 0x7f84e4153660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56124/0 (socket says 172.21.15.88:56124) 2023-10-18T16:09:14.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:14.946+0000 7f84e37fe700 1 -- 172.21.15.88:0/1191007717 learned_addr learned my addr 172.21.15.88:0/1191007717 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:09:14.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:14.946+0000 7f84e3fff700 1 --2- 172.21.15.88:0/1191007717 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f84e40c6a60 0x7f84e40c6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:14.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:14.946+0000 7f84e37fe700 1 -- 172.21.15.88:0/1191007717 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f84e40c6a60 msgr2=0x7f84e40c6e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:09:14.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:14.946+0000 7f84e37fe700 1 --2- 172.21.15.88:0/1191007717 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f84e40c6a60 0x7f84e40c6e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:14.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:14.946+0000 7f84e37fe700 1 -- 172.21.15.88:0/1191007717 >> v1:172.21.15.151:6789/0 conn(0x7f84e4153b50 legacy=0x7f84e40c2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:09:14.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:14.946+0000 7f84e37fe700 1 -- 172.21.15.88:0/1191007717 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84e40c2f70 con 0x7f84e4063d70 2023-10-18T16:09:14.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:14.946+0000 7f84e37fe700 1 --2- 172.21.15.88:0/1191007717 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84e4063d70 0x7f84e4153660 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f84d8006fd0 tx=0x7f84d80090f0).ready entity=mon.0 client_cookie=92693aca2b82be1d server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:09:14.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:14.947+0000 7f84e2ffd700 1 -- 172.21.15.88:0/1191007717 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d800b600 con 0x7f84e4063d70 2023-10-18T16:09:14.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:14.947+0000 7f84e2ffd700 1 -- 172.21.15.88:0/1191007717 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84d800b760 con 0x7f84e4063d70 2023-10-18T16:09:14.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:14.947+0000 7f84e2ffd700 1 -- 172.21.15.88:0/1191007717 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d800bae0 con 0x7f84e4063d70 2023-10-18T16:09:14.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:14.947+0000 7f84e94d7700 1 -- 172.21.15.88:0/1191007717 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84e4063d70 msgr2=0x7f84e4153660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:14.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:14.947+0000 7f84e94d7700 1 --2- 172.21.15.88:0/1191007717 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84e4063d70 0x7f84e4153660 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f84d8006fd0 tx=0x7f84d80090f0).stop 2023-10-18T16:09:14.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:14.947+0000 7f84e94d7700 1 -- 172.21.15.88:0/1191007717 shutdown_connections 2023-10-18T16:09:14.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:14.947+0000 7f84e94d7700 1 --2- 172.21.15.88:0/1191007717 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f84e40c6a60 0x7f84e40c6e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:14.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:14.947+0000 7f84e94d7700 1 --2- 172.21.15.88:0/1191007717 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84e4063d70 0x7f84e4153660 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:14.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:14.947+0000 7f84e94d7700 1 -- 172.21.15.88:0/1191007717 >> 172.21.15.88:0/1191007717 conn(0x7f84e405e050 msgr2=0x7f84e405e460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:09:14.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:14.947+0000 7f84e94d7700 1 -- 172.21.15.88:0/1191007717 shutdown_connections 2023-10-18T16:09:14.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:14.947+0000 7f84e94d7700 1 -- 172.21.15.88:0/1191007717 wait complete. 2023-10-18T16:09:14.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:14.948+0000 7f84e94d7700 1 Processor -- start 2023-10-18T16:09:14.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:14.949+0000 7f84e94d7700 1 -- start start 2023-10-18T16:09:14.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:14.949+0000 7f84e94d7700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84e4063d70 0x7f84e4164380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:14.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:14.949+0000 7f84e94d7700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f84e40c6a60 0x7f84e415e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:14.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:14.949+0000 7f84e94d7700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f84e4153b50 0x7f84e414f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:14.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:14.949+0000 7f84e94d7700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e4160d30 con 0x7f84e4063d70 2023-10-18T16:09:14.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:14.949+0000 7f84e94d7700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e4160420 con 0x7f84e4153b50 2023-10-18T16:09:14.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:14.949+0000 7f84e94d7700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84e4152580 con 0x7f84e40c6a60 2023-10-18T16:09:14.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:14.949+0000 7f84e3fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84e4063d70 0x7f84e4164380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:14.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:14.949+0000 7f84e3fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84e4063d70 0x7f84e4164380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56130/0 (socket says 172.21.15.88:56130) 2023-10-18T16:09:14.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:14.949+0000 7f84e3fff700 1 -- 172.21.15.88:0/2639998949 learned_addr learned my addr 172.21.15.88:0/2639998949 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:09:14.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:14.950+0000 7f84e37fe700 1 --2- 172.21.15.88:0/2639998949 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f84e40c6a60 0x7f84e415e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:14.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:14.950+0000 7f84e3fff700 1 -- 172.21.15.88:0/2639998949 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f84e40c6a60 msgr2=0x7f84e415e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:14.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:14.950+0000 7f84e3fff700 1 --2- 172.21.15.88:0/2639998949 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f84e40c6a60 0x7f84e415e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:14.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:14.950+0000 7f84e8cd6700 1 --2- 172.21.15.88:0/2639998949 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f84e4153b50 0x7f84e414f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:14.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:14.950+0000 7f84e3fff700 1 -- 172.21.15.88:0/2639998949 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f84e4153b50 msgr2=0x7f84e414f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:14.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:14.950+0000 7f84e3fff700 1 --2- 172.21.15.88:0/2639998949 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f84e4153b50 0x7f84e414f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:14.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:14.950+0000 7f84e3fff700 1 -- 172.21.15.88:0/2639998949 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84d8009370 con 0x7f84e4063d70 2023-10-18T16:09:14.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:14.950+0000 7f84e8cd6700 1 --2- 172.21.15.88:0/2639998949 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f84e4153b50 0x7f84e414f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-10-18T16:09:14.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:14.950+0000 7f84e3fff700 1 --2- 172.21.15.88:0/2639998949 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84e4063d70 0x7f84e4164380 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f84d400c850 tx=0x7f84d4008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:09:14.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:14.950+0000 7f84cbfff700 1 -- 172.21.15.88:0/2639998949 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d4010580 con 0x7f84e4063d70 2023-10-18T16:09:14.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:14.950+0000 7f84cbfff700 1 -- 172.21.15.88:0/2639998949 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84d40106e0 con 0x7f84e4063d70 2023-10-18T16:09:14.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:14.950+0000 7f84e94d7700 1 -- 172.21.15.88:0/2639998949 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84e4152860 con 0x7f84e4063d70 2023-10-18T16:09:14.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:14.950+0000 7f84cbfff700 1 -- 172.21.15.88:0/2639998949 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d4010a60 con 0x7f84e4063d70 2023-10-18T16:09:14.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:14.950+0000 7f84e94d7700 1 -- 172.21.15.88:0/2639998949 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84e41609c0 con 0x7f84e4063d70 2023-10-18T16:09:14.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:14.951+0000 7f84e94d7700 1 -- 172.21.15.88:0/2639998949 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84e4084340 con 0x7f84e4063d70 2023-10-18T16:09:14.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:14.951+0000 7f84cbfff700 1 -- 172.21.15.88:0/2639998949 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f84d401b460 con 0x7f84e4063d70 2023-10-18T16:09:14.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:14.951+0000 7f84cbfff700 1 --2- 172.21.15.88:0/2639998949 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f84d005f150 0x7f84d0061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:14.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:14.952+0000 7f84cbfff700 1 -- 172.21.15.88:0/2639998949 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6553+0+0 (secure 0 0 0) 0x7f84d407a310 con 0x7f84e4063d70 2023-10-18T16:09:14.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:14.952+0000 7f84e37fe700 1 --2- 172.21.15.88:0/2639998949 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f84d005f150 0x7f84d0061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:14.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:14.952+0000 7f84e37fe700 1 --2- 172.21.15.88:0/2639998949 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f84d005f150 0x7f84d0061610 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f84d800b3c0 tx=0x7f84d8017000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:09:14.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:14.956+0000 7f84cbfff700 1 -- 172.21.15.88:0/2639998949 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f84d404b3d0 con 0x7f84e4063d70 2023-10-18T16:09:15.160 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:15.156+0000 7f84e94d7700 1 -- 172.21.15.88:0/2639998949 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f84e406aff0 con 0x7f84e4063d70 2023-10-18T16:09:15.161 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:15.157+0000 7f84cbfff700 1 -- 172.21.15.88:0/2639998949 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7f84d404e9f0 con 0x7f84e4063d70 2023-10-18T16:09:15.161 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:09:15.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:15.159+0000 7f84e94d7700 1 -- 172.21.15.88:0/2639998949 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f84d005f150 msgr2=0x7f84d0061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:15.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:15.159+0000 7f84e94d7700 1 --2- 172.21.15.88:0/2639998949 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f84d005f150 0x7f84d0061610 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f84d800b3c0 tx=0x7f84d8017000).stop 2023-10-18T16:09:15.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:15.159+0000 7f84e94d7700 1 -- 172.21.15.88:0/2639998949 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84e4063d70 msgr2=0x7f84e4164380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:15.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:15.159+0000 7f84e94d7700 1 --2- 172.21.15.88:0/2639998949 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84e4063d70 0x7f84e4164380 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f84d400c850 tx=0x7f84d4008040).stop 2023-10-18T16:09:15.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:15.159+0000 7f84e94d7700 1 -- 172.21.15.88:0/2639998949 shutdown_connections 2023-10-18T16:09:15.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:15.159+0000 7f84e94d7700 1 --2- 172.21.15.88:0/2639998949 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f84d005f150 0x7f84d0061610 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:15.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:15.159+0000 7f84e94d7700 1 --2- 172.21.15.88:0/2639998949 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f84e4153b50 0x7f84e414f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:15.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:15.159+0000 7f84e94d7700 1 --2- 172.21.15.88:0/2639998949 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84e4063d70 0x7f84e4164380 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:15.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:15.159+0000 7f84e94d7700 1 --2- 172.21.15.88:0/2639998949 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f84e40c6a60 0x7f84e415e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:15.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:15.159+0000 7f84e94d7700 1 -- 172.21.15.88:0/2639998949 >> 172.21.15.88:0/2639998949 conn(0x7f84e405e050 msgr2=0x7f84e414ca70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:09:15.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:15.159+0000 7f84e94d7700 1 -- 172.21.15.88:0/2639998949 shutdown_connections 2023-10-18T16:09:15.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:15.159+0000 7f84e94d7700 1 -- 172.21.15.88:0/2639998949 wait complete. 2023-10-18T16:09:15.174 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:09:16.743 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~26s 2023-10-18T16:09:16.743 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~7s 2023-10-18T16:09:16.744 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~87s 2023-10-18T16:09:16.744 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.f is failed for ~40s 2023-10-18T16:09:18.176 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:09:18.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.297+0000 7f94e6976700 1 Processor -- start 2023-10-18T16:09:18.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.297+0000 7f94e6976700 1 -- start start 2023-10-18T16:09:18.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.298+0000 7f94e6976700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f94e0063d70 0x7f94e00c6e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:18.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.298+0000 7f94e6976700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94e0151020 con 0x7f94e00c2400 2023-10-18T16:09:18.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.298+0000 7f94e6976700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94e0150b90 con 0x7f94e00c6a60 2023-10-18T16:09:18.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.298+0000 7f94e6976700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94e0150090 con 0x7f94e0063d70 2023-10-18T16:09:18.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.298+0000 7f94e5173700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f94e0063d70 0x7f94e00c6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:18.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.298+0000 7f94e6175700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f94e00c2400 0x7f94e00c3300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:48332/0 (socket says 172.21.15.88:48332) 2023-10-18T16:09:18.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.298+0000 7f94e6175700 1 -- 172.21.15.88:0/2531442022 learned_addr learned my addr 172.21.15.88:0/2531442022 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:09:18.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.298+0000 7f94e5173700 1 -- 172.21.15.88:0/2531442022 >> v1:172.21.15.151:6789/0 conn(0x7f94e00c6a60 legacy=0x7f94e0153660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:09:18.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.298+0000 7f94e4972700 1 -- 172.21.15.88:0/2531442022 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2688073534 0 0) 0x7f94d0001110 con 0x7f94e00c2400 2023-10-18T16:09:18.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.298+0000 7f94e5173700 1 -- 172.21.15.88:0/2531442022 >> v1:172.21.15.88:6789/0 conn(0x7f94e00c2400 legacy=0x7f94e00c3300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:18.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.298+0000 7f94e5173700 1 -- 172.21.15.88:0/2531442022 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94e00c2f70 con 0x7f94e0063d70 2023-10-18T16:09:18.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.298+0000 7f94e5173700 1 --2- 172.21.15.88:0/2531442022 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f94e0063d70 0x7f94e00c6e40 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f94d4009760 tx=0x7f94d4009060).ready entity=mon.0 client_cookie=67ff9cdddd2a638 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:09:18.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.299+0000 7f94e4972700 1 -- 172.21.15.88:0/2531442022 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94d400b580 con 0x7f94e0063d70 2023-10-18T16:09:18.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.299+0000 7f94e4972700 1 -- 172.21.15.88:0/2531442022 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94d400b6e0 con 0x7f94e0063d70 2023-10-18T16:09:18.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.299+0000 7f94e4972700 1 -- 172.21.15.88:0/2531442022 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94d400ba60 con 0x7f94e0063d70 2023-10-18T16:09:18.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.299+0000 7f94e6976700 1 -- 172.21.15.88:0/2531442022 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f94e0063d70 msgr2=0x7f94e00c6e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:18.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.299+0000 7f94e6976700 1 --2- 172.21.15.88:0/2531442022 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f94e0063d70 0x7f94e00c6e40 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f94d4009760 tx=0x7f94d4009060).stop 2023-10-18T16:09:18.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.299+0000 7f94e6976700 1 -- 172.21.15.88:0/2531442022 shutdown_connections 2023-10-18T16:09:18.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.299+0000 7f94e6976700 1 --2- 172.21.15.88:0/2531442022 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f94e0063d70 0x7f94e00c6e40 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:18.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.299+0000 7f94e6976700 1 -- 172.21.15.88:0/2531442022 >> 172.21.15.88:0/2531442022 conn(0x7f94e005e050 msgr2=0x7f94e005e460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:09:18.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.299+0000 7f94e6976700 1 -- 172.21.15.88:0/2531442022 shutdown_connections 2023-10-18T16:09:18.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.299+0000 7f94e6976700 1 -- 172.21.15.88:0/2531442022 wait complete. 2023-10-18T16:09:18.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.300+0000 7f94e6976700 1 Processor -- start 2023-10-18T16:09:18.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.300+0000 7f94e6976700 1 -- start start 2023-10-18T16:09:18.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.301+0000 7f94e6976700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f94e0063d70 0x7f94e00ba810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:18.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.301+0000 7f94e6976700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f94e00c2400 0x7f94e014d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:18.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.301+0000 7f94e6976700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f94e00c6a60 0x7f94e0163910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:18.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.301+0000 7f94e6976700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94e0162570 con 0x7f94e00c2400 2023-10-18T16:09:18.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.301+0000 7f94e6976700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94e0162850 con 0x7f94e00c6a60 2023-10-18T16:09:18.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.301+0000 7f94e6976700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94e0162b30 con 0x7f94e0063d70 2023-10-18T16:09:18.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.301+0000 7f94e5173700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f94e00c2400 0x7f94e014d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:18.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.301+0000 7f94e5173700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f94e00c2400 0x7f94e014d170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:39064/0 (socket says 172.21.15.88:39064) 2023-10-18T16:09:18.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.301+0000 7f94e5173700 1 -- 172.21.15.88:0/777644927 learned_addr learned my addr 172.21.15.88:0/777644927 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:09:18.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.301+0000 7f94e5974700 1 --2- 172.21.15.88:0/777644927 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f94e0063d70 0x7f94e00ba810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:18.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.301+0000 7f94e6175700 1 --2- 172.21.15.88:0/777644927 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f94e00c6a60 0x7f94e0163910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:18.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.301+0000 7f94e5173700 1 -- 172.21.15.88:0/777644927 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f94e0063d70 msgr2=0x7f94e00ba810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:18.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.301+0000 7f94e5173700 1 --2- 172.21.15.88:0/777644927 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f94e0063d70 0x7f94e00ba810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:18.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.301+0000 7f94e5173700 1 -- 172.21.15.88:0/777644927 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f94e00c6a60 msgr2=0x7f94e0163910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:18.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.301+0000 7f94e5173700 1 --2- 172.21.15.88:0/777644927 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f94e00c6a60 0x7f94e0163910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:18.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.301+0000 7f94e5173700 1 -- 172.21.15.88:0/777644927 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94d40092a0 con 0x7f94e00c2400 2023-10-18T16:09:18.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.301+0000 7f94e5974700 1 --2- 172.21.15.88:0/777644927 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f94e0063d70 0x7f94e00ba810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:09:18.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.301+0000 7f94e6175700 1 --2- 172.21.15.88:0/777644927 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f94e00c6a60 0x7f94e0163910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:09:18.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.301+0000 7f94e5173700 1 --2- 172.21.15.88:0/777644927 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f94e00c2400 0x7f94e014d170 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f94d4009eb0 tx=0x7f94d4017000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:09:18.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.302+0000 7f94ddffb700 1 -- 172.21.15.88:0/777644927 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94d400ba60 con 0x7f94e00c2400 2023-10-18T16:09:18.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.302+0000 7f94ddffb700 1 -- 172.21.15.88:0/777644927 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94d4000c90 con 0x7f94e00c2400 2023-10-18T16:09:18.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.302+0000 7f94e6976700 1 -- 172.21.15.88:0/777644927 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94e0162e10 con 0x7f94e00c2400 2023-10-18T16:09:18.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.302+0000 7f94ddffb700 1 -- 172.21.15.88:0/777644927 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94d401b640 con 0x7f94e00c2400 2023-10-18T16:09:18.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.302+0000 7f94e6976700 1 -- 172.21.15.88:0/777644927 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94e014e7f0 con 0x7f94e00c2400 2023-10-18T16:09:18.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.303+0000 7f94e6976700 1 -- 172.21.15.88:0/777644927 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94e0084340 con 0x7f94e00c2400 2023-10-18T16:09:18.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.303+0000 7f94ddffb700 1 -- 172.21.15.88:0/777644927 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f94d401ba30 con 0x7f94e00c2400 2023-10-18T16:09:18.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.303+0000 7f94ddffb700 1 --2- 172.21.15.88:0/777644927 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f94cc05f150 0x7f94cc061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:18.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.304+0000 7f94ddffb700 1 -- 172.21.15.88:0/777644927 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6553+0+0 (secure 0 0 0) 0x7f94d407bd40 con 0x7f94e00c2400 2023-10-18T16:09:18.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.304+0000 7f94e5974700 1 --2- 172.21.15.88:0/777644927 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f94cc05f150 0x7f94cc061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:18.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.304+0000 7f94e5974700 1 --2- 172.21.15.88:0/777644927 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f94cc05f150 0x7f94cc061610 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f94d8001d90 tx=0x7f94d8009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:09:18.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.308+0000 7f94ddffb700 1 -- 172.21.15.88:0/777644927 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f94d404ce60 con 0x7f94e00c2400 2023-10-18T16:09:18.434 INFO:tasks.ceph.mds.c.smithi088.stderr:terminate called after throwing an instance of 'boost::wrapexcept' 2023-10-18T16:09:18.435 INFO:tasks.ceph.mds.c.smithi088.stderr: what(): boost::bad_get: failed value get using boost::get 2023-10-18T16:09:18.435 INFO:tasks.ceph.mds.c.smithi088.stderr:*** Caught signal (Aborted) ** 2023-10-18T16:09:18.435 INFO:tasks.ceph.mds.c.smithi088.stderr: in thread 7f6f2cdf2700 thread_name:ms_dispatch 2023-10-18T16:09:18.437 INFO:tasks.ceph.mds.c.smithi088.stderr: ceph version 16.2.14-100-gb4ddd52c (b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5) pacific (stable) 2023-10-18T16:09:18.437 INFO:tasks.ceph.mds.c.smithi088.stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f6f345f5b20] 2023-10-18T16:09:18.437 INFO:tasks.ceph.mds.c.smithi088.stderr: 2: gsignal() 2023-10-18T16:09:18.438 INFO:tasks.ceph.mds.c.smithi088.stderr: 3: abort() 2023-10-18T16:09:18.438 INFO:tasks.ceph.mds.c.smithi088.stderr: 4: /lib64/libstdc++.so.6(+0x9009b) [0x7f6f339f609b] 2023-10-18T16:09:18.438 INFO:tasks.ceph.mds.c.smithi088.stderr: 5: /lib64/libstdc++.so.6(+0x9653c) [0x7f6f339fc53c] 2023-10-18T16:09:18.439 INFO:tasks.ceph.mds.c.smithi088.stderr: 6: /lib64/libstdc++.so.6(+0x96597) [0x7f6f339fc597] 2023-10-18T16:09:18.439 INFO:tasks.ceph.mds.c.smithi088.stderr: 7: /lib64/libstdc++.so.6(+0x967f8) [0x7f6f339fc7f8] 2023-10-18T16:09:18.440 INFO:tasks.ceph.mds.c.smithi088.stderr: 8: ceph-mds(+0x212b43) [0x55a997141b43] 2023-10-18T16:09:18.440 INFO:tasks.ceph.mds.c.smithi088.stderr: 9: (Option::size_t const md_config_t::get_val(ConfigValues const&, std::basic_string_view >) const+0x5c) [0x55a9971b689c] 2023-10-18T16:09:18.440 INFO:tasks.ceph.mds.c.smithi088.stderr: 10: (SessionMap::SessionMap(MDSRank*)+0x2c2) [0x55a9973fe6a2] 2023-10-18T16:09:18.441 INFO:tasks.ceph.mds.c.smithi088.stderr: 11: (MDSRank::MDSRank(int, std::__cxx11::basic_string, std::allocator >, ceph::fair_mutex&, std::shared_ptr&, CommonSafeTimer&, Beacon&, std::unique_ptr >&, Messenger*, MonClient*, MgrClient*, Context*, Context*, boost::asio::io_context&)+0x214) [0x55a99711b724] 2023-10-18T16:09:18.441 INFO:tasks.ceph.mds.c.smithi088.stderr: 12: (MDSRankDispatcher::MDSRankDispatcher(int, std::__cxx11::basic_string, std::allocator >, ceph::fair_mutex&, std::shared_ptr&, CommonSafeTimer&, Beacon&, std::unique_ptr >&, Messenger*, MonClient*, MgrClient*, Context*, Context*, boost::asio::io_context&)+0x101) [0x55a99711d181] 2023-10-18T16:09:18.441 INFO:tasks.ceph.mds.c.smithi088.stderr: 13: (MDSDaemon::handle_mds_map(boost::intrusive_ptr const&)+0x1608) [0x55a9970ed348] 2023-10-18T16:09:18.442 INFO:tasks.ceph.mds.c.smithi088.stderr: 14: (MDSDaemon::handle_core_message(boost::intrusive_ptr const&)+0xcd) [0x55a9970f044d] 2023-10-18T16:09:18.442 INFO:tasks.ceph.mds.c.smithi088.stderr: 15: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0xc3) [0x55a9970f0fa3] 2023-10-18T16:09:18.442 INFO:tasks.ceph.mds.c.smithi088.stderr: 16: (DispatchQueue::entry()+0x126a) [0x7f6f3584f4ea] 2023-10-18T16:09:18.443 INFO:tasks.ceph.mds.c.smithi088.stderr: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6f35903d51] 2023-10-18T16:09:18.443 INFO:tasks.ceph.mds.c.smithi088.stderr: 18: /lib64/libpthread.so.0(+0x814a) [0x7f6f345eb14a] 2023-10-18T16:09:18.444 INFO:tasks.ceph.mds.c.smithi088.stderr: 19: clone() 2023-10-18T16:09:18.444 INFO:tasks.ceph.mds.c.smithi088.stderr:2023-10-18T16:09:18.432+0000 7f6f2cdf2700 -1 *** Caught signal (Aborted) ** 2023-10-18T16:09:18.444 INFO:tasks.ceph.mds.c.smithi088.stderr: in thread 7f6f2cdf2700 thread_name:ms_dispatch 2023-10-18T16:09:18.445 INFO:tasks.ceph.mds.c.smithi088.stderr: 2023-10-18T16:09:18.445 INFO:tasks.ceph.mds.c.smithi088.stderr: ceph version 16.2.14-100-gb4ddd52c (b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5) pacific (stable) 2023-10-18T16:09:18.445 INFO:tasks.ceph.mds.c.smithi088.stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f6f345f5b20] 2023-10-18T16:09:18.446 INFO:tasks.ceph.mds.c.smithi088.stderr: 2: gsignal() 2023-10-18T16:09:18.446 INFO:tasks.ceph.mds.c.smithi088.stderr: 3: abort() 2023-10-18T16:09:18.446 INFO:tasks.ceph.mds.c.smithi088.stderr: 4: /lib64/libstdc++.so.6(+0x9009b) [0x7f6f339f609b] 2023-10-18T16:09:18.447 INFO:tasks.ceph.mds.c.smithi088.stderr: 5: /lib64/libstdc++.so.6(+0x9653c) [0x7f6f339fc53c] 2023-10-18T16:09:18.447 INFO:tasks.ceph.mds.c.smithi088.stderr: 6: /lib64/libstdc++.so.6(+0x96597) [0x7f6f339fc597] 2023-10-18T16:09:18.447 INFO:tasks.ceph.mds.c.smithi088.stderr: 7: /lib64/libstdc++.so.6(+0x967f8) [0x7f6f339fc7f8] 2023-10-18T16:09:18.448 INFO:tasks.ceph.mds.c.smithi088.stderr: 8: ceph-mds(+0x212b43) [0x55a997141b43] 2023-10-18T16:09:18.448 INFO:tasks.ceph.mds.c.smithi088.stderr: 9: (Option::size_t const md_config_t::get_val(ConfigValues const&, std::basic_string_view >) const+0x5c) [0x55a9971b689c] 2023-10-18T16:09:18.448 INFO:tasks.ceph.mds.c.smithi088.stderr: 10: (SessionMap::SessionMap(MDSRank*)+0x2c2) [0x55a9973fe6a2] 2023-10-18T16:09:18.449 INFO:tasks.ceph.mds.c.smithi088.stderr: 11: (MDSRank::MDSRank(int, std::__cxx11::basic_string, std::allocator >, ceph::fair_mutex&, std::shared_ptr&, CommonSafeTimer&, Beacon&, std::unique_ptr >&, Messenger*, MonClient*, MgrClient*, Context*, Context*, boost::asio::io_context&)+0x214) [0x55a99711b724] 2023-10-18T16:09:18.449 INFO:tasks.ceph.mds.c.smithi088.stderr: 12: (MDSRankDispatcher::MDSRankDispatcher(int, std::__cxx11::basic_string, std::allocator >, ceph::fair_mutex&, std::shared_ptr&, CommonSafeTimer&, Beacon&, std::unique_ptr >&, Messenger*, MonClient*, MgrClient*, Context*, Context*, boost::asio::io_context&)+0x101) [0x55a99711d181] 2023-10-18T16:09:18.449 INFO:tasks.ceph.mds.c.smithi088.stderr: 13: (MDSDaemon::handle_mds_map(boost::intrusive_ptr const&)+0x1608) [0x55a9970ed348] 2023-10-18T16:09:18.450 INFO:tasks.ceph.mds.c.smithi088.stderr: 14: (MDSDaemon::handle_core_message(boost::intrusive_ptr const&)+0xcd) [0x55a9970f044d] 2023-10-18T16:09:18.450 INFO:tasks.ceph.mds.c.smithi088.stderr: 15: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0xc3) [0x55a9970f0fa3] 2023-10-18T16:09:18.450 INFO:tasks.ceph.mds.c.smithi088.stderr: 16: (DispatchQueue::entry()+0x126a) [0x7f6f3584f4ea] 2023-10-18T16:09:18.453 INFO:tasks.ceph.mds.c.smithi088.stderr: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6f35903d51] 2023-10-18T16:09:18.453 INFO:tasks.ceph.mds.c.smithi088.stderr: 18: /lib64/libpthread.so.0(+0x814a) [0x7f6f345eb14a] 2023-10-18T16:09:18.454 INFO:tasks.ceph.mds.c.smithi088.stderr: 19: clone() 2023-10-18T16:09:18.454 INFO:tasks.ceph.mds.c.smithi088.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-10-18T16:09:18.454 INFO:tasks.ceph.mds.c.smithi088.stderr: 2023-10-18T16:09:18.455 INFO:tasks.ceph.mds.c.smithi088.stderr: 0> 2023-10-18T16:09:18.432+0000 7f6f2cdf2700 -1 *** Caught signal (Aborted) ** 2023-10-18T16:09:18.455 INFO:tasks.ceph.mds.c.smithi088.stderr: in thread 7f6f2cdf2700 thread_name:ms_dispatch 2023-10-18T16:09:18.456 INFO:tasks.ceph.mds.c.smithi088.stderr: 2023-10-18T16:09:18.456 INFO:tasks.ceph.mds.c.smithi088.stderr: ceph version 16.2.14-100-gb4ddd52c (b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5) pacific (stable) 2023-10-18T16:09:18.457 INFO:tasks.ceph.mds.c.smithi088.stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f6f345f5b20] 2023-10-18T16:09:18.457 INFO:tasks.ceph.mds.c.smithi088.stderr: 2: gsignal() 2023-10-18T16:09:18.457 INFO:tasks.ceph.mds.c.smithi088.stderr: 3: abort() 2023-10-18T16:09:18.458 INFO:tasks.ceph.mds.c.smithi088.stderr: 4: /lib64/libstdc++.so.6(+0x9009b) [0x7f6f339f609b] 2023-10-18T16:09:18.458 INFO:tasks.ceph.mds.c.smithi088.stderr: 5: /lib64/libstdc++.so.6(+0x9653c) [0x7f6f339fc53c] 2023-10-18T16:09:18.458 INFO:tasks.ceph.mds.c.smithi088.stderr: 6: /lib64/libstdc++.so.6(+0x96597) [0x7f6f339fc597] 2023-10-18T16:09:18.458 INFO:tasks.ceph.mds.c.smithi088.stderr: 7: /lib64/libstdc++.so.6(+0x967f8) [0x7f6f339fc7f8] 2023-10-18T16:09:18.459 INFO:tasks.ceph.mds.c.smithi088.stderr: 8: ceph-mds(+0x212b43) [0x55a997141b43] 2023-10-18T16:09:18.459 INFO:tasks.ceph.mds.c.smithi088.stderr: 9: (Option::size_t const md_config_t::get_val(ConfigValues const&, std::basic_string_view >) const+0x5c) [0x55a9971b689c] 2023-10-18T16:09:18.459 INFO:tasks.ceph.mds.c.smithi088.stderr: 10: (SessionMap::SessionMap(MDSRank*)+0x2c2) [0x55a9973fe6a2] 2023-10-18T16:09:18.459 INFO:tasks.ceph.mds.c.smithi088.stderr: 11: (MDSRank::MDSRank(int, std::__cxx11::basic_string, std::allocator >, ceph::fair_mutex&, std::shared_ptr&, CommonSafeTimer&, Beacon&, std::unique_ptr >&, Messenger*, MonClient*, MgrClient*, Context*, Context*, boost::asio::io_context&)+0x214) [0x55a99711b724] 2023-10-18T16:09:18.460 INFO:tasks.ceph.mds.c.smithi088.stderr: 12: (MDSRankDispatcher::MDSRankDispatcher(int, std::__cxx11::basic_string, std::allocator >, ceph::fair_mutex&, std::shared_ptr&, CommonSafeTimer&, Beacon&, std::unique_ptr >&, Messenger*, MonClient*, MgrClient*, Context*, Context*, boost::asio::io_context&)+0x101) [0x55a99711d181] 2023-10-18T16:09:18.460 INFO:tasks.ceph.mds.c.smithi088.stderr: 13: (MDSDaemon::handle_mds_map(boost::intrusive_ptr const&)+0x1608) [0x55a9970ed348] 2023-10-18T16:09:18.460 INFO:tasks.ceph.mds.c.smithi088.stderr: 14: (MDSDaemon::handle_core_message(boost::intrusive_ptr const&)+0xcd) [0x55a9970f044d] 2023-10-18T16:09:18.461 INFO:tasks.ceph.mds.c.smithi088.stderr: 15: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0xc3) [0x55a9970f0fa3] 2023-10-18T16:09:18.461 INFO:tasks.ceph.mds.c.smithi088.stderr: 16: (DispatchQueue::entry()+0x126a) [0x7f6f3584f4ea] 2023-10-18T16:09:18.461 INFO:tasks.ceph.mds.c.smithi088.stderr: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6f35903d51] 2023-10-18T16:09:18.461 INFO:tasks.ceph.mds.c.smithi088.stderr: 18: /lib64/libpthread.so.0(+0x814a) [0x7f6f345eb14a] 2023-10-18T16:09:18.462 INFO:tasks.ceph.mds.c.smithi088.stderr: 19: clone() 2023-10-18T16:09:18.462 INFO:tasks.ceph.mds.c.smithi088.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-10-18T16:09:18.462 INFO:tasks.ceph.mds.c.smithi088.stderr: 2023-10-18T16:09:18.463 INFO:tasks.ceph.mds.c.smithi088.stderr: 0> 2023-10-18T16:09:18.432+0000 7f6f2cdf2700 -1 *** Caught signal (Aborted) ** 2023-10-18T16:09:18.463 INFO:tasks.ceph.mds.c.smithi088.stderr: in thread 7f6f2cdf2700 thread_name:ms_dispatch 2023-10-18T16:09:18.463 INFO:tasks.ceph.mds.c.smithi088.stderr: 2023-10-18T16:09:18.464 INFO:tasks.ceph.mds.c.smithi088.stderr: ceph version 16.2.14-100-gb4ddd52c (b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5) pacific (stable) 2023-10-18T16:09:18.464 INFO:tasks.ceph.mds.c.smithi088.stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f6f345f5b20] 2023-10-18T16:09:18.464 INFO:tasks.ceph.mds.c.smithi088.stderr: 2: gsignal() 2023-10-18T16:09:18.465 INFO:tasks.ceph.mds.c.smithi088.stderr: 3: abort() 2023-10-18T16:09:18.465 INFO:tasks.ceph.mds.c.smithi088.stderr: 4: /lib64/libstdc++.so.6(+0x9009b) [0x7f6f339f609b] 2023-10-18T16:09:18.465 INFO:tasks.ceph.mds.c.smithi088.stderr: 5: /lib64/libstdc++.so.6(+0x9653c) [0x7f6f339fc53c] 2023-10-18T16:09:18.466 INFO:tasks.ceph.mds.c.smithi088.stderr: 6: /lib64/libstdc++.so.6(+0x96597) [0x7f6f339fc597] 2023-10-18T16:09:18.466 INFO:tasks.ceph.mds.c.smithi088.stderr: 7: /lib64/libstdc++.so.6(+0x967f8) [0x7f6f339fc7f8] 2023-10-18T16:09:18.466 INFO:tasks.ceph.mds.c.smithi088.stderr: 8: ceph-mds(+0x212b43) [0x55a997141b43] 2023-10-18T16:09:18.467 INFO:tasks.ceph.mds.c.smithi088.stderr: 9: (Option::size_t const md_config_t::get_val(ConfigValues const&, std::basic_string_view >) const+0x5c) [0x55a9971b689c] 2023-10-18T16:09:18.467 INFO:tasks.ceph.mds.c.smithi088.stderr: 10: (SessionMap::SessionMap(MDSRank*)+0x2c2) [0x55a9973fe6a2] 2023-10-18T16:09:18.467 INFO:tasks.ceph.mds.c.smithi088.stderr: 11: (MDSRank::MDSRank(int, std::__cxx11::basic_string, std::allocator >, ceph::fair_mutex&, std::shared_ptr&, CommonSafeTimer&, Beacon&, std::unique_ptr >&, Messenger*, MonClient*, MgrClient*, Context*, Context*, boost::asio::io_context&)+0x214) [0x55a99711b724] 2023-10-18T16:09:18.468 INFO:tasks.ceph.mds.c.smithi088.stderr: 12: (MDSRankDispatcher::MDSRankDispatcher(int, std::__cxx11::basic_string, std::allocator >, ceph::fair_mutex&, std::shared_ptr&, CommonSafeTimer&, Beacon&, std::unique_ptr >&, Messenger*, MonClient*, MgrClient*, Context*, Context*, boost::asio::io_context&)+0x101) [0x55a99711d181] 2023-10-18T16:09:18.468 INFO:tasks.ceph.mds.c.smithi088.stderr: 13: (MDSDaemon::handle_mds_map(boost::intrusive_ptr const&)+0x1608) [0x55a9970ed348] 2023-10-18T16:09:18.468 INFO:tasks.ceph.mds.c.smithi088.stderr: 14: (MDSDaemon::handle_core_message(boost::intrusive_ptr const&)+0xcd) [0x55a9970f044d] 2023-10-18T16:09:18.469 INFO:tasks.ceph.mds.c.smithi088.stderr: 15: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0xc3) [0x55a9970f0fa3] 2023-10-18T16:09:18.469 INFO:tasks.ceph.mds.c.smithi088.stderr: 16: (DispatchQueue::entry()+0x126a) [0x7f6f3584f4ea] 2023-10-18T16:09:18.470 INFO:tasks.ceph.mds.c.smithi088.stderr: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6f35903d51] 2023-10-18T16:09:18.470 INFO:tasks.ceph.mds.c.smithi088.stderr: 18: /lib64/libpthread.so.0(+0x814a) [0x7f6f345eb14a] 2023-10-18T16:09:18.470 INFO:tasks.ceph.mds.c.smithi088.stderr: 19: clone() 2023-10-18T16:09:18.470 INFO:tasks.ceph.mds.c.smithi088.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-10-18T16:09:18.471 INFO:tasks.ceph.mds.c.smithi088.stderr: 2023-10-18T16:09:18.515 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.511+0000 7f94e6976700 1 -- 172.21.15.88:0/777644927 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f94e0046c20 con 0x7f94e00c2400 2023-10-18T16:09:18.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.512+0000 7f94ddffb700 1 -- 172.21.15.88:0/777644927 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7f94d4050480 con 0x7f94e00c2400 2023-10-18T16:09:18.517 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:09:18.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.514+0000 7f94e6976700 1 -- 172.21.15.88:0/777644927 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f94cc05f150 msgr2=0x7f94cc061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:18.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.514+0000 7f94e6976700 1 --2- 172.21.15.88:0/777644927 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f94cc05f150 0x7f94cc061610 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f94d8001d90 tx=0x7f94d8009040).stop 2023-10-18T16:09:18.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.514+0000 7f94e6976700 1 -- 172.21.15.88:0/777644927 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f94e00c2400 msgr2=0x7f94e014d170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:18.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.514+0000 7f94e6976700 1 --2- 172.21.15.88:0/777644927 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f94e00c2400 0x7f94e014d170 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f94d4009eb0 tx=0x7f94d4017000).stop 2023-10-18T16:09:18.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.514+0000 7f94e6976700 1 -- 172.21.15.88:0/777644927 shutdown_connections 2023-10-18T16:09:18.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.514+0000 7f94e6976700 1 --2- 172.21.15.88:0/777644927 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f94cc05f150 0x7f94cc061610 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:18.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.514+0000 7f94e6976700 1 --2- 172.21.15.88:0/777644927 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f94e00c6a60 0x7f94e0163910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:18.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.514+0000 7f94e6976700 1 --2- 172.21.15.88:0/777644927 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f94e0063d70 0x7f94e00ba810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:18.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.514+0000 7f94e6976700 1 --2- 172.21.15.88:0/777644927 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f94e00c2400 0x7f94e014d170 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:18.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.514+0000 7f94e6976700 1 -- 172.21.15.88:0/777644927 >> 172.21.15.88:0/777644927 conn(0x7f94e005e050 msgr2=0x7f94e014ad90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:09:18.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.514+0000 7f94e6976700 1 -- 172.21.15.88:0/777644927 shutdown_connections 2023-10-18T16:09:18.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:18.514+0000 7f94e6976700 1 -- 172.21.15.88:0/777644927 wait complete. 2023-10-18T16:09:18.529 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:09:18.553 INFO:tasks.ceph.mds.c.smithi088.stderr:daemon-helper: command crashed with signal 6 2023-10-18T16:09:21.530 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:09:21.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.653+0000 7f02ece35700 1 Processor -- start 2023-10-18T16:09:21.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.653+0000 7f02ece35700 1 -- start start 2023-10-18T16:09:21.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.654+0000 7f02ece35700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f02e80c6a60 0x7f02e80c6e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:21.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.654+0000 7f02ece35700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f02e80c2400 0x7f02e80c2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:21.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.654+0000 7f02ece35700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02e8151020 con 0x7f02e8063d70 2023-10-18T16:09:21.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.654+0000 7f02ece35700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02e8150090 con 0x7f02e80c6a60 2023-10-18T16:09:21.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.654+0000 7f02ece35700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02e80c2d80 con 0x7f02e80c2400 2023-10-18T16:09:21.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.654+0000 7f02dffff700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f02e8063d70 0x7f02e8153660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:48344/0 (socket says 172.21.15.88:48344) 2023-10-18T16:09:21.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.654+0000 7f02dffff700 1 -- 172.21.15.88:0/1823081327 learned_addr learned my addr 172.21.15.88:0/1823081327 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:09:21.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.654+0000 7f02e7fff700 1 --2- 172.21.15.88:0/1823081327 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f02e80c2400 0x7f02e80c2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:21.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.654+0000 7f02e77fe700 1 --2- 172.21.15.88:0/1823081327 >> v2:172.21.15.151:3300/0 conn(0x7f02e80c6a60 0x7f02e80c6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:21.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.655+0000 7f02e6ffd700 1 -- 172.21.15.88:0/1823081327 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4205865627 0 0) 0x7f02d0001110 con 0x7f02e8063d70 2023-10-18T16:09:21.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.655+0000 7f02e6ffd700 1 -- 172.21.15.88:0/1823081327 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02d40033f0 con 0x7f02e8063d70 2023-10-18T16:09:21.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.655+0000 7f02e6ffd700 1 -- 172.21.15.88:0/1823081327 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 793230114 0 0) 0x7f02d40033f0 con 0x7f02e8063d70 2023-10-18T16:09:21.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.655+0000 7f02e77fe700 1 -- 172.21.15.88:0/1823081327 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f02e80c2400 msgr2=0x7f02e80c2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:09:21.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.655+0000 7f02e77fe700 1 --2- 172.21.15.88:0/1823081327 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f02e80c2400 0x7f02e80c2890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:21.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.655+0000 7f02e77fe700 1 -- 172.21.15.88:0/1823081327 >> v1:172.21.15.88:6789/0 conn(0x7f02e8063d70 legacy=0x7f02e8153660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:21.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.655+0000 7f02e77fe700 1 -- 172.21.15.88:0/1823081327 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02e8150b90 con 0x7f02e80c6a60 2023-10-18T16:09:21.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.655+0000 7f02e7fff700 1 --2- 172.21.15.88:0/1823081327 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f02e80c2400 0x7f02e80c2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2023-10-18T16:09:21.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.655+0000 7f02e77fe700 1 --2- 172.21.15.88:0/1823081327 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02e80c6a60 0x7f02e80c6e40 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f02d8009790 tx=0x7f02d8009060).ready entity=mon.1 client_cookie=61d4950b0fb962c6 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:09:21.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.655+0000 7f02e6ffd700 1 -- 172.21.15.88:0/1823081327 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d0001600 con 0x7f02e80c6a60 2023-10-18T16:09:21.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.655+0000 7f02e6ffd700 1 -- 172.21.15.88:0/1823081327 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02d8011400 con 0x7f02e80c6a60 2023-10-18T16:09:21.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.655+0000 7f02e6ffd700 1 -- 172.21.15.88:0/1823081327 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d8011780 con 0x7f02e80c6a60 2023-10-18T16:09:21.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.656+0000 7f02ece35700 1 -- 172.21.15.88:0/1823081327 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02e80c6a60 msgr2=0x7f02e80c6e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:21.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.656+0000 7f02ece35700 1 --2- 172.21.15.88:0/1823081327 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02e80c6a60 0x7f02e80c6e40 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f02d8009790 tx=0x7f02d8009060).stop 2023-10-18T16:09:21.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.656+0000 7f02ece35700 1 -- 172.21.15.88:0/1823081327 shutdown_connections 2023-10-18T16:09:21.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.656+0000 7f02ece35700 1 --2- 172.21.15.88:0/1823081327 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f02e80c2400 0x7f02e80c2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:21.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.656+0000 7f02ece35700 1 --2- 172.21.15.88:0/1823081327 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02e80c6a60 0x7f02e80c6e40 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:21.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.656+0000 7f02ece35700 1 -- 172.21.15.88:0/1823081327 >> 172.21.15.88:0/1823081327 conn(0x7f02e805e050 msgr2=0x7f02e805e460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:09:21.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.656+0000 7f02ece35700 1 -- 172.21.15.88:0/1823081327 shutdown_connections 2023-10-18T16:09:21.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.656+0000 7f02ece35700 1 -- 172.21.15.88:0/1823081327 wait complete. 2023-10-18T16:09:21.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.657+0000 7f02ece35700 1 Processor -- start 2023-10-18T16:09:21.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.657+0000 7f02ece35700 1 -- start start 2023-10-18T16:09:21.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.658+0000 7f02ece35700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02e8063d70 0x7f02e8160d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:21.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.658+0000 7f02ece35700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f02e80c2400 0x7f02e80c5f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:21.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.658+0000 7f02ece35700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02e80c6a60 0x7f02e8152ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:21.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.658+0000 7f02ece35700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02e8161250 con 0x7f02e8063d70 2023-10-18T16:09:21.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.658+0000 7f02ece35700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02e8152810 con 0x7f02e80c6a60 2023-10-18T16:09:21.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.658+0000 7f02ece35700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02e8152530 con 0x7f02e80c2400 2023-10-18T16:09:21.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.658+0000 7f02e77fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02e8063d70 0x7f02e8160d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:21.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.658+0000 7f02e77fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02e8063d70 0x7f02e8160d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:39068/0 (socket says 172.21.15.88:39068) 2023-10-18T16:09:21.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.658+0000 7f02e77fe700 1 -- 172.21.15.88:0/2070975443 learned_addr learned my addr 172.21.15.88:0/2070975443 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:09:21.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.658+0000 7f02e7fff700 1 --2- 172.21.15.88:0/2070975443 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02e80c6a60 0x7f02e8152ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:21.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.658+0000 7f02dffff700 1 --2- 172.21.15.88:0/2070975443 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f02e80c2400 0x7f02e80c5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:21.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.658+0000 7f02e77fe700 1 -- 172.21.15.88:0/2070975443 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f02e80c2400 msgr2=0x7f02e80c5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:21.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.658+0000 7f02e77fe700 1 --2- 172.21.15.88:0/2070975443 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f02e80c2400 0x7f02e80c5f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:21.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.658+0000 7f02e77fe700 1 -- 172.21.15.88:0/2070975443 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02e80c6a60 msgr2=0x7f02e8152ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:21.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.658+0000 7f02e77fe700 1 --2- 172.21.15.88:0/2070975443 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02e80c6a60 0x7f02e8152ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:21.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.658+0000 7f02e77fe700 1 -- 172.21.15.88:0/2070975443 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02e0009a20 con 0x7f02e8063d70 2023-10-18T16:09:21.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.658+0000 7f02dffff700 1 --2- 172.21.15.88:0/2070975443 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f02e80c2400 0x7f02e80c5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:09:21.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.658+0000 7f02e77fe700 1 --2- 172.21.15.88:0/2070975443 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02e8063d70 0x7f02e8160d60 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f02e805ec40 tx=0x7f02d8016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:09:21.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.658+0000 7f02df7fe700 1 -- 172.21.15.88:0/2070975443 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d8010920 con 0x7f02e8063d70 2023-10-18T16:09:21.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.658+0000 7f02df7fe700 1 -- 172.21.15.88:0/2070975443 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02d8010a80 con 0x7f02e8063d70 2023-10-18T16:09:21.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.658+0000 7f02ece35700 1 -- 172.21.15.88:0/2070975443 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02d80092a0 con 0x7f02e8063d70 2023-10-18T16:09:21.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.658+0000 7f02df7fe700 1 -- 172.21.15.88:0/2070975443 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d8010e20 con 0x7f02e8063d70 2023-10-18T16:09:21.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.658+0000 7f02ece35700 1 -- 172.21.15.88:0/2070975443 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02e8153b30 con 0x7f02e8063d70 2023-10-18T16:09:21.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.659+0000 7f02ece35700 1 -- 172.21.15.88:0/2070975443 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02e8152af0 con 0x7f02e8063d70 2023-10-18T16:09:21.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.660+0000 7f02df7fe700 1 -- 172.21.15.88:0/2070975443 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f02d801d640 con 0x7f02e8063d70 2023-10-18T16:09:21.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.660+0000 7f02df7fe700 1 --2- 172.21.15.88:0/2070975443 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f02c805f150 0x7f02c8061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:21.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.660+0000 7f02df7fe700 1 -- 172.21.15.88:0/2070975443 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6639+0+0 (secure 0 0 0) 0x7f02d807ba30 con 0x7f02e8063d70 2023-10-18T16:09:21.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.660+0000 7f02dffff700 1 --2- 172.21.15.88:0/2070975443 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f02c805f150 0x7f02c8061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:21.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.661+0000 7f02dffff700 1 --2- 172.21.15.88:0/2070975443 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f02c805f150 0x7f02c8061610 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f02e814d110 tx=0x7f02d0001650).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:09:21.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.664+0000 7f02df7fe700 1 -- 172.21.15.88:0/2070975443 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f02d804cb00 con 0x7f02e8063d70 2023-10-18T16:09:21.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.865+0000 7f02ece35700 1 -- 172.21.15.88:0/2070975443 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f02e8152670 con 0x7f02e8063d70 2023-10-18T16:09:21.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.865+0000 7f02df7fe700 1 -- 172.21.15.88:0/2070975443 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7f02e8152670 con 0x7f02e8063d70 2023-10-18T16:09:21.869 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:09:21.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.867+0000 7f02ece35700 1 -- 172.21.15.88:0/2070975443 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f02c805f150 msgr2=0x7f02c8061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:21.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.867+0000 7f02ece35700 1 --2- 172.21.15.88:0/2070975443 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f02c805f150 0x7f02c8061610 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f02e814d110 tx=0x7f02d0001650).stop 2023-10-18T16:09:21.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.867+0000 7f02ece35700 1 -- 172.21.15.88:0/2070975443 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02e8063d70 msgr2=0x7f02e8160d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:21.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.867+0000 7f02ece35700 1 --2- 172.21.15.88:0/2070975443 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02e8063d70 0x7f02e8160d60 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f02e805ec40 tx=0x7f02d8016000).stop 2023-10-18T16:09:21.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.867+0000 7f02ece35700 1 -- 172.21.15.88:0/2070975443 shutdown_connections 2023-10-18T16:09:21.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.867+0000 7f02ece35700 1 --2- 172.21.15.88:0/2070975443 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f02c805f150 0x7f02c8061610 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:21.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.867+0000 7f02ece35700 1 --2- 172.21.15.88:0/2070975443 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02e80c6a60 0x7f02e8152ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:21.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.867+0000 7f02ece35700 1 --2- 172.21.15.88:0/2070975443 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02e8063d70 0x7f02e8160d60 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:21.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.867+0000 7f02ece35700 1 --2- 172.21.15.88:0/2070975443 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f02e80c2400 0x7f02e80c5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:21.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.867+0000 7f02ece35700 1 -- 172.21.15.88:0/2070975443 >> 172.21.15.88:0/2070975443 conn(0x7f02e805e050 msgr2=0x7f02e814f520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:09:21.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.867+0000 7f02ece35700 1 -- 172.21.15.88:0/2070975443 shutdown_connections 2023-10-18T16:09:21.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:21.867+0000 7f02ece35700 1 -- 172.21.15.88:0/2070975443 wait complete. 2023-10-18T16:09:21.882 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:09:23.155 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~0s 2023-10-18T16:09:23.155 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~33s 2023-10-18T16:09:23.155 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~13s 2023-10-18T16:09:23.155 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~94s 2023-10-18T16:09:23.156 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.f is failed for ~46s 2023-10-18T16:09:24.883 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:09:25.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.003+0000 7fa3244de700 1 Processor -- start 2023-10-18T16:09:25.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.003+0000 7fa3244de700 1 -- start start 2023-10-18T16:09:25.010 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.004+0000 7fa3244de700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa31c1533f0 0x7fa31c151ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:25.010 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.004+0000 7fa3244de700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa31c0a9670 con 0x7fa31c153010 2023-10-18T16:09:25.010 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.004+0000 7fa3244de700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa31c151a50 con 0x7fa31c1523d0 2023-10-18T16:09:25.010 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.004+0000 7fa3244de700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa31c14df50 con 0x7fa31c1533f0 2023-10-18T16:09:25.010 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.005+0000 7fa321a79700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa31c1533f0 0x7fa31c151ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:25.011 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.005+0000 7fa32227a700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fa31c153010 0x7fa31c064d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:48356/0 (socket says 172.21.15.88:48356) 2023-10-18T16:09:25.011 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.005+0000 7fa32227a700 1 -- 172.21.15.88:0/3171597384 learned_addr learned my addr 172.21.15.88:0/3171597384 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:09:25.011 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.005+0000 7fa321a79700 1 --2- 172.21.15.88:0/3171597384 >> v2:172.21.15.88:3300/0 conn(0x7fa31c1533f0 0x7fa31c151ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:39076/0 (socket says 172.21.15.88:39076) 2023-10-18T16:09:25.011 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.005+0000 7fa321a79700 1 -- 172.21.15.88:0/3171597384 >> v1:172.21.15.151:6789/0 conn(0x7fa31c1523d0 legacy=0x7fa31c0c36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:09:25.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.005+0000 7fa321a79700 1 -- 172.21.15.88:0/3171597384 >> v1:172.21.15.88:6789/0 conn(0x7fa31c153010 legacy=0x7fa31c064d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:25.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.005+0000 7fa321278700 1 -- 172.21.15.88:0/3171597384 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3075732860 0 0) 0x7fa31c0a9670 con 0x7fa31c153010 2023-10-18T16:09:25.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.005+0000 7fa321a79700 1 -- 172.21.15.88:0/3171597384 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa31c0a8af0 con 0x7fa31c1533f0 2023-10-18T16:09:25.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.005+0000 7fa321a79700 1 --2- 172.21.15.88:0/3171597384 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa31c1533f0 0x7fa31c151ee0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7fa30c009760 tx=0x7fa30c009060).ready entity=mon.0 client_cookie=1dd98a38729d3e1a server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:09:25.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.005+0000 7fa321278700 1 -- 172.21.15.88:0/3171597384 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa30c009120 con 0x7fa31c1533f0 2023-10-18T16:09:25.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.005+0000 7fa321278700 1 -- 172.21.15.88:0/3171597384 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa30c00b580 con 0x7fa31c1533f0 2023-10-18T16:09:25.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.005+0000 7fa321278700 1 -- 172.21.15.88:0/3171597384 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa30c00b900 con 0x7fa31c1533f0 2023-10-18T16:09:25.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.006+0000 7fa3244de700 1 -- 172.21.15.88:0/3171597384 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa31c1533f0 msgr2=0x7fa31c151ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:25.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.006+0000 7fa3244de700 1 --2- 172.21.15.88:0/3171597384 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa31c1533f0 0x7fa31c151ee0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7fa30c009760 tx=0x7fa30c009060).stop 2023-10-18T16:09:25.014 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.006+0000 7fa3244de700 1 -- 172.21.15.88:0/3171597384 shutdown_connections 2023-10-18T16:09:25.014 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.006+0000 7fa3244de700 1 --2- 172.21.15.88:0/3171597384 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa31c1533f0 0x7fa31c151ee0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:25.014 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.006+0000 7fa3244de700 1 -- 172.21.15.88:0/3171597384 >> 172.21.15.88:0/3171597384 conn(0x7fa31c069d80 msgr2=0x7fa31c06a190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:09:25.014 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.006+0000 7fa3244de700 1 -- 172.21.15.88:0/3171597384 shutdown_connections 2023-10-18T16:09:25.014 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.006+0000 7fa3244de700 1 -- 172.21.15.88:0/3171597384 wait complete. 2023-10-18T16:09:25.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.007+0000 7fa3244de700 1 Processor -- start 2023-10-18T16:09:25.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.007+0000 7fa3244de700 1 -- start start 2023-10-18T16:09:25.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.008+0000 7fa3244de700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa31c1523d0 0x7fa31c0bcab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:25.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.008+0000 7fa3244de700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa31c153010 0x7fa31c163f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:25.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.008+0000 7fa3244de700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa31c1533f0 0x7fa31c1538f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:25.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.008+0000 7fa3244de700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa31c0bd800 con 0x7fa31c1533f0 2023-10-18T16:09:25.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.008+0000 7fa3244de700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa31c161c90 con 0x7fa31c153010 2023-10-18T16:09:25.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.008+0000 7fa3244de700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa31c164460 con 0x7fa31c1523d0 2023-10-18T16:09:25.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.008+0000 7fa322a7b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa31c1533f0 0x7fa31c1538f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:25.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.008+0000 7fa322a7b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa31c1533f0 0x7fa31c1538f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:39090/0 (socket says 172.21.15.88:39090) 2023-10-18T16:09:25.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.008+0000 7fa322a7b700 1 -- 172.21.15.88:0/2550830982 learned_addr learned my addr 172.21.15.88:0/2550830982 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:09:25.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.008+0000 7fa321a79700 1 --2- 172.21.15.88:0/2550830982 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa31c153010 0x7fa31c163f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:25.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.008+0000 7fa32227a700 1 --2- 172.21.15.88:0/2550830982 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa31c1523d0 0x7fa31c0bcab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:25.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.008+0000 7fa322a7b700 1 -- 172.21.15.88:0/2550830982 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa31c1523d0 msgr2=0x7fa31c0bcab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:25.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.008+0000 7fa322a7b700 1 --2- 172.21.15.88:0/2550830982 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa31c1523d0 0x7fa31c0bcab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:25.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.008+0000 7fa322a7b700 1 -- 172.21.15.88:0/2550830982 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa31c153010 msgr2=0x7fa31c163f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:25.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.008+0000 7fa322a7b700 1 --2- 172.21.15.88:0/2550830982 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa31c153010 0x7fa31c163f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:25.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.008+0000 7fa322a7b700 1 -- 172.21.15.88:0/2550830982 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa30c0092a0 con 0x7fa31c1533f0 2023-10-18T16:09:25.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.008+0000 7fa322a7b700 1 --2- 172.21.15.88:0/2550830982 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa31c1533f0 0x7fa31c1538f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7fa31400c440 tx=0x7fa314007d70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:09:25.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.008+0000 7fa3127fc700 1 -- 172.21.15.88:0/2550830982 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3140123f0 con 0x7fa31c1533f0 2023-10-18T16:09:25.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.008+0000 7fa3127fc700 1 -- 172.21.15.88:0/2550830982 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa314001e90 con 0x7fa31c1533f0 2023-10-18T16:09:25.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.008+0000 7fa3244de700 1 -- 172.21.15.88:0/2550830982 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa31c0bd240 con 0x7fa31c1533f0 2023-10-18T16:09:25.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.008+0000 7fa3127fc700 1 -- 172.21.15.88:0/2550830982 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa314012760 con 0x7fa31c1533f0 2023-10-18T16:09:25.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.008+0000 7fa3244de700 1 -- 172.21.15.88:0/2550830982 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa31c161640 con 0x7fa31c1533f0 2023-10-18T16:09:25.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.009+0000 7fa3244de700 1 -- 172.21.15.88:0/2550830982 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa31c1614a0 con 0x7fa31c1533f0 2023-10-18T16:09:25.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.010+0000 7fa3127fc700 1 -- 172.21.15.88:0/2550830982 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7fa314012b10 con 0x7fa31c1533f0 2023-10-18T16:09:25.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.010+0000 7fa3127fc700 1 --2- 172.21.15.88:0/2550830982 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fa30005f150 0x7fa300061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:25.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.010+0000 7fa3127fc700 1 -- 172.21.15.88:0/2550830982 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6639+0+0 (secure 0 0 0) 0x7fa3140796d0 con 0x7fa31c1533f0 2023-10-18T16:09:25.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.010+0000 7fa32227a700 1 --2- 172.21.15.88:0/2550830982 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fa30005f150 0x7fa300061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:25.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.011+0000 7fa32227a700 1 --2- 172.21.15.88:0/2550830982 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fa30005f150 0x7fa300061610 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fa318001560 tx=0x7fa318007f10).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:09:25.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.014+0000 7fa3127fc700 1 -- 172.21.15.88:0/2550830982 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7fa314047ac0 con 0x7fa31c1533f0 2023-10-18T16:09:25.215 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.212+0000 7fa3244de700 1 -- 172.21.15.88:0/2550830982 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fa31c154200 con 0x7fa31c1533f0 2023-10-18T16:09:25.216 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.212+0000 7fa3127fc700 1 -- 172.21.15.88:0/2550830982 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7fa31c1614a0 con 0x7fa31c1533f0 2023-10-18T16:09:25.216 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:09:25.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.214+0000 7fa3244de700 1 -- 172.21.15.88:0/2550830982 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fa30005f150 msgr2=0x7fa300061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:25.218 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.214+0000 7fa3244de700 1 --2- 172.21.15.88:0/2550830982 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fa30005f150 0x7fa300061610 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fa318001560 tx=0x7fa318007f10).stop 2023-10-18T16:09:25.218 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.214+0000 7fa3244de700 1 -- 172.21.15.88:0/2550830982 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa31c1533f0 msgr2=0x7fa31c1538f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:25.218 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.214+0000 7fa3244de700 1 --2- 172.21.15.88:0/2550830982 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa31c1533f0 0x7fa31c1538f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7fa31400c440 tx=0x7fa314007d70).stop 2023-10-18T16:09:25.218 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.214+0000 7fa3244de700 1 -- 172.21.15.88:0/2550830982 shutdown_connections 2023-10-18T16:09:25.218 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.214+0000 7fa3244de700 1 --2- 172.21.15.88:0/2550830982 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fa30005f150 0x7fa300061610 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:25.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.214+0000 7fa3244de700 1 --2- 172.21.15.88:0/2550830982 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa31c1533f0 0x7fa31c1538f0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:25.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.214+0000 7fa3244de700 1 --2- 172.21.15.88:0/2550830982 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa31c1523d0 0x7fa31c0bcab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:25.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.214+0000 7fa3244de700 1 --2- 172.21.15.88:0/2550830982 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa31c153010 0x7fa31c163f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:25.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.214+0000 7fa3244de700 1 -- 172.21.15.88:0/2550830982 >> 172.21.15.88:0/2550830982 conn(0x7fa31c069d80 msgr2=0x7fa31c0a9b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:09:25.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.214+0000 7fa3244de700 1 -- 172.21.15.88:0/2550830982 shutdown_connections 2023-10-18T16:09:25.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:25.214+0000 7fa3244de700 1 -- 172.21.15.88:0/2550830982 wait complete. 2023-10-18T16:09:25.229 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:09:28.230 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:09:28.355 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.351+0000 7f5bd6920700 1 Processor -- start 2023-10-18T16:09:28.355 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.351+0000 7f5bd6920700 1 -- start start 2023-10-18T16:09:28.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.352+0000 7f5bd6920700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5bd0065360 0x7f5bd015bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:28.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.352+0000 7f5bd6920700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bd0064150 con 0x7f5bd00a9050 2023-10-18T16:09:28.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.352+0000 7f5bd6920700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bd0153a40 con 0x7f5bd014d8c0 2023-10-18T16:09:28.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.352+0000 7f5bd6920700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bd0063e70 con 0x7f5bd0065360 2023-10-18T16:09:28.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.352+0000 7f5bd4ebd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5bd0065360 0x7f5bd015bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:28.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.352+0000 7f5bcf7fe700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f5bd00a9050 0x7f5bd0153ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:33960/0 (socket says 172.21.15.88:33960) 2023-10-18T16:09:28.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.352+0000 7f5bcf7fe700 1 -- 172.21.15.88:0/1357803837 learned_addr learned my addr 172.21.15.88:0/1357803837 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:09:28.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.353+0000 7f5bd4ebd700 1 -- 172.21.15.88:0/1357803837 >> v1:172.21.15.151:6789/0 conn(0x7f5bd014d8c0 legacy=0x7f5bd0064440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:09:28.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.353+0000 7f5bd4ebd700 1 -- 172.21.15.88:0/1357803837 >> v1:172.21.15.88:6789/0 conn(0x7f5bd00a9050 legacy=0x7f5bd0153ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:28.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.353+0000 7f5bd4ebd700 1 -- 172.21.15.88:0/1357803837 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bd00c72c0 con 0x7f5bd0065360 2023-10-18T16:09:28.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.353+0000 7f5bceffd700 1 -- 172.21.15.88:0/1357803837 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1299182987 0 0) 0x7f5bd0064150 con 0x7f5bd00a9050 2023-10-18T16:09:28.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.353+0000 7f5bd4ebd700 1 --2- 172.21.15.88:0/1357803837 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5bd0065360 0x7f5bd015bcc0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f5bc8009e60 tx=0x7f5bc8009780).ready entity=mon.0 client_cookie=bbb6276878d4ab71 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:09:28.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.353+0000 7f5bceffd700 1 -- 172.21.15.88:0/1357803837 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bc800e070 con 0x7f5bd0065360 2023-10-18T16:09:28.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.353+0000 7f5bceffd700 1 -- 172.21.15.88:0/1357803837 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bc800bbf0 con 0x7f5bd0065360 2023-10-18T16:09:28.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.353+0000 7f5bceffd700 1 -- 172.21.15.88:0/1357803837 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bc8012410 con 0x7f5bd0065360 2023-10-18T16:09:28.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.353+0000 7f5bd6920700 1 -- 172.21.15.88:0/1357803837 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5bd0065360 msgr2=0x7f5bd015bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:28.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.353+0000 7f5bd6920700 1 --2- 172.21.15.88:0/1357803837 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5bd0065360 0x7f5bd015bcc0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f5bc8009e60 tx=0x7f5bc8009780).stop 2023-10-18T16:09:28.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.353+0000 7f5bd6920700 1 -- 172.21.15.88:0/1357803837 shutdown_connections 2023-10-18T16:09:28.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.353+0000 7f5bd6920700 1 --2- 172.21.15.88:0/1357803837 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5bd0065360 0x7f5bd015bcc0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:28.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.354+0000 7f5bd6920700 1 -- 172.21.15.88:0/1357803837 >> 172.21.15.88:0/1357803837 conn(0x7f5bd006a810 msgr2=0x7f5bd00c6a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:09:28.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.354+0000 7f5bd6920700 1 -- 172.21.15.88:0/1357803837 shutdown_connections 2023-10-18T16:09:28.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.354+0000 7f5bd6920700 1 -- 172.21.15.88:0/1357803837 wait complete. 2023-10-18T16:09:28.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.355+0000 7f5bd6920700 1 Processor -- start 2023-10-18T16:09:28.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.355+0000 7f5bd6920700 1 -- start start 2023-10-18T16:09:28.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.355+0000 7f5bd6920700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5bd0065360 0x7f5bd00ac670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:28.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.355+0000 7f5bd6920700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5bd00a9050 0x7f5bd0160a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:28.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.356+0000 7f5bd6920700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5bd014d8c0 0x7f5bd00bc760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:28.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.356+0000 7f5bd6920700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd0084be0 con 0x7f5bd014d8c0 2023-10-18T16:09:28.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.356+0000 7f5bd6920700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd00bbd10 con 0x7f5bd0065360 2023-10-18T16:09:28.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.356+0000 7f5bd6920700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd00abb20 con 0x7f5bd00a9050 2023-10-18T16:09:28.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.356+0000 7f5bd4ebd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5bd014d8c0 0x7f5bd00bc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:28.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.356+0000 7f5bd4ebd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5bd014d8c0 0x7f5bd00bc760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41344/0 (socket says 172.21.15.88:41344) 2023-10-18T16:09:28.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.356+0000 7f5bd4ebd700 1 -- 172.21.15.88:0/907051645 learned_addr learned my addr 172.21.15.88:0/907051645 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:09:28.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.356+0000 7f5bcf7fe700 1 --2- 172.21.15.88:0/907051645 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5bd00a9050 0x7f5bd0160a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:28.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.356+0000 7f5bcffff700 1 --2- 172.21.15.88:0/907051645 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5bd0065360 0x7f5bd00ac670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:28.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.356+0000 7f5bd4ebd700 1 -- 172.21.15.88:0/907051645 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5bd00a9050 msgr2=0x7f5bd0160a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:28.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.356+0000 7f5bd4ebd700 1 --2- 172.21.15.88:0/907051645 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5bd00a9050 0x7f5bd0160a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:28.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.356+0000 7f5bd4ebd700 1 -- 172.21.15.88:0/907051645 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5bd0065360 msgr2=0x7f5bd00ac670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:28.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.356+0000 7f5bd4ebd700 1 --2- 172.21.15.88:0/907051645 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5bd0065360 0x7f5bd00ac670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:28.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.356+0000 7f5bd4ebd700 1 -- 172.21.15.88:0/907051645 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bc8009a20 con 0x7f5bd014d8c0 2023-10-18T16:09:28.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.356+0000 7f5bcf7fe700 1 --2- 172.21.15.88:0/907051645 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5bd00a9050 0x7f5bd0160a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-10-18T16:09:28.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.356+0000 7f5bcffff700 1 --2- 172.21.15.88:0/907051645 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5bd0065360 0x7f5bd00ac670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-10-18T16:09:28.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.356+0000 7f5bd4ebd700 1 --2- 172.21.15.88:0/907051645 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5bd014d8c0 0x7f5bd00bc760 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f5bc800b6c0 tx=0x7f5bc8009780).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:09:28.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.356+0000 7f5bb7fff700 1 -- 172.21.15.88:0/907051645 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bc80116d0 con 0x7f5bd014d8c0 2023-10-18T16:09:28.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.356+0000 7f5bb7fff700 1 -- 172.21.15.88:0/907051645 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bc8011830 con 0x7f5bd014d8c0 2023-10-18T16:09:28.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.356+0000 7f5bd6920700 1 -- 172.21.15.88:0/907051645 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bd00abe00 con 0x7f5bd014d8c0 2023-10-18T16:09:28.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.356+0000 7f5bb7fff700 1 -- 172.21.15.88:0/907051645 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bc8011bf0 con 0x7f5bd014d8c0 2023-10-18T16:09:28.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.356+0000 7f5bd6920700 1 -- 172.21.15.88:0/907051645 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bd00bbf60 con 0x7f5bd014d8c0 2023-10-18T16:09:28.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.357+0000 7f5bd6920700 1 -- 172.21.15.88:0/907051645 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bd006c4e0 con 0x7f5bd014d8c0 2023-10-18T16:09:28.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.358+0000 7f5bb7fff700 1 -- 172.21.15.88:0/907051645 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f5bc8027460 con 0x7f5bd014d8c0 2023-10-18T16:09:28.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.358+0000 7f5bb7fff700 1 --2- 172.21.15.88:0/907051645 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f5bbc05f150 0x7f5bbc061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:28.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.358+0000 7f5bb7fff700 1 -- 172.21.15.88:0/907051645 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6639+0+0 (secure 0 0 0) 0x7f5bc807c6d0 con 0x7f5bd014d8c0 2023-10-18T16:09:28.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.358+0000 7f5bcffff700 1 --2- 172.21.15.88:0/907051645 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f5bbc05f150 0x7f5bbc061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:28.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.358+0000 7f5bcffff700 1 --2- 172.21.15.88:0/907051645 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f5bbc05f150 0x7f5bbc061610 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f5bc000a820 tx=0x7f5bc0008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:09:28.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.362+0000 7f5bb7fff700 1 -- 172.21.15.88:0/907051645 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f5bc804d740 con 0x7f5bd014d8c0 2023-10-18T16:09:28.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.565+0000 7f5bd6920700 1 -- 172.21.15.88:0/907051645 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f5bd0065140 con 0x7f5bd014d8c0 2023-10-18T16:09:28.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.565+0000 7f5bb7fff700 1 -- 172.21.15.88:0/907051645 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7f5bc8050d60 con 0x7f5bd014d8c0 2023-10-18T16:09:28.569 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:09:28.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.567+0000 7f5bd6920700 1 -- 172.21.15.88:0/907051645 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f5bbc05f150 msgr2=0x7f5bbc061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:28.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.567+0000 7f5bd6920700 1 --2- 172.21.15.88:0/907051645 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f5bbc05f150 0x7f5bbc061610 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f5bc000a820 tx=0x7f5bc0008040).stop 2023-10-18T16:09:28.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.567+0000 7f5bd6920700 1 -- 172.21.15.88:0/907051645 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5bd014d8c0 msgr2=0x7f5bd00bc760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:28.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.567+0000 7f5bd6920700 1 --2- 172.21.15.88:0/907051645 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5bd014d8c0 0x7f5bd00bc760 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f5bc800b6c0 tx=0x7f5bc8009780).stop 2023-10-18T16:09:28.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.567+0000 7f5bd6920700 1 -- 172.21.15.88:0/907051645 shutdown_connections 2023-10-18T16:09:28.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.567+0000 7f5bd6920700 1 --2- 172.21.15.88:0/907051645 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f5bbc05f150 0x7f5bbc061610 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:28.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.567+0000 7f5bd6920700 1 --2- 172.21.15.88:0/907051645 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5bd014d8c0 0x7f5bd00bc760 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:28.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.567+0000 7f5bd6920700 1 --2- 172.21.15.88:0/907051645 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5bd0065360 0x7f5bd00ac670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:28.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.567+0000 7f5bd6920700 1 --2- 172.21.15.88:0/907051645 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5bd00a9050 0x7f5bd0160a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:28.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.567+0000 7f5bd6920700 1 -- 172.21.15.88:0/907051645 >> 172.21.15.88:0/907051645 conn(0x7f5bd006a810 msgr2=0x7f5bd014cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:09:28.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.567+0000 7f5bd6920700 1 -- 172.21.15.88:0/907051645 shutdown_connections 2023-10-18T16:09:28.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:28.567+0000 7f5bd6920700 1 -- 172.21.15.88:0/907051645 wait complete. 2023-10-18T16:09:28.582 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:09:29.570 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~6s 2023-10-18T16:09:29.571 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~39s 2023-10-18T16:09:29.571 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~19s 2023-10-18T16:09:29.571 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~100s 2023-10-18T16:09:29.571 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.f is failed for ~53s 2023-10-18T16:09:31.584 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:09:31.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.705+0000 7fbb1c9c5700 1 Processor -- start 2023-10-18T16:09:31.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.705+0000 7fbb1c9c5700 1 -- start start 2023-10-18T16:09:31.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.706+0000 7fbb1c9c5700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fbb180c6a60 0x7fbb180c6e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:31.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.706+0000 7fbb1c9c5700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbb18063d70 0x7fbb18153660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:31.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.706+0000 7fbb1c9c5700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb18151020 con 0x7fbb18153b50 2023-10-18T16:09:31.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.706+0000 7fbb1c9c5700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb18150090 con 0x7fbb180c6a60 2023-10-18T16:09:31.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.706+0000 7fbb1c9c5700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb18150b90 con 0x7fbb18063d70 2023-10-18T16:09:31.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.706+0000 7fbb16ffd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbb18063d70 0x7fbb18153660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:31.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.706+0000 7fbb177fe700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fbb180c6a60 0x7fbb180c6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:31.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.706+0000 7fbb16ffd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbb18063d70 0x7fbb18153660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.88:59380/0 (socket says 172.21.15.88:59380) 2023-10-18T16:09:31.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.706+0000 7fbb16ffd700 1 -- 172.21.15.88:0/1413149078 learned_addr learned my addr 172.21.15.88:0/1413149078 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:09:31.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.707+0000 7fbb177fe700 1 -- 172.21.15.88:0/1413149078 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbb18063d70 msgr2=0x7fbb18153660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:09:31.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.707+0000 7fbb177fe700 1 --2- 172.21.15.88:0/1413149078 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbb18063d70 0x7fbb18153660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:31.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.707+0000 7fbb177fe700 1 -- 172.21.15.88:0/1413149078 >> v1:172.21.15.151:6789/0 conn(0x7fbb18153b50 legacy=0x7fbb180c2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:31.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.707+0000 7fbb177fe700 1 -- 172.21.15.88:0/1413149078 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb180c2f70 con 0x7fbb180c6a60 2023-10-18T16:09:31.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.707+0000 7fbb16ffd700 1 --2- 172.21.15.88:0/1413149078 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbb18063d70 0x7fbb18153660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2023-10-18T16:09:31.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.707+0000 7fbb177fe700 1 --2- 172.21.15.88:0/1413149078 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbb180c6a60 0x7fbb180c6e40 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fbb08009e60 tx=0x7fbb08009780).ready entity=mon.1 client_cookie=29d8dc201154af79 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:09:31.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.707+0000 7fbb167fc700 1 -- 172.21.15.88:0/1413149078 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb0800e070 con 0x7fbb180c6a60 2023-10-18T16:09:31.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.707+0000 7fbb167fc700 1 -- 172.21.15.88:0/1413149078 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb0800bb60 con 0x7fbb180c6a60 2023-10-18T16:09:31.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.707+0000 7fbb167fc700 1 -- 172.21.15.88:0/1413149078 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb08012410 con 0x7fbb180c6a60 2023-10-18T16:09:31.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.707+0000 7fbb1c9c5700 1 -- 172.21.15.88:0/1413149078 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbb180c6a60 msgr2=0x7fbb180c6e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:31.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.707+0000 7fbb1c9c5700 1 --2- 172.21.15.88:0/1413149078 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbb180c6a60 0x7fbb180c6e40 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fbb08009e60 tx=0x7fbb08009780).stop 2023-10-18T16:09:31.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.707+0000 7fbb1c9c5700 1 -- 172.21.15.88:0/1413149078 shutdown_connections 2023-10-18T16:09:31.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.707+0000 7fbb1c9c5700 1 --2- 172.21.15.88:0/1413149078 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbb180c6a60 0x7fbb180c6e40 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:31.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.707+0000 7fbb1c9c5700 1 --2- 172.21.15.88:0/1413149078 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbb18063d70 0x7fbb18153660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:31.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.707+0000 7fbb1c9c5700 1 -- 172.21.15.88:0/1413149078 >> 172.21.15.88:0/1413149078 conn(0x7fbb1805e050 msgr2=0x7fbb1805e460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:09:31.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.708+0000 7fbb1c9c5700 1 -- 172.21.15.88:0/1413149078 shutdown_connections 2023-10-18T16:09:31.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.708+0000 7fbb1c9c5700 1 -- 172.21.15.88:0/1413149078 wait complete. 2023-10-18T16:09:31.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.709+0000 7fbb1c9c5700 1 Processor -- start 2023-10-18T16:09:31.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.709+0000 7fbb1c9c5700 1 -- start start 2023-10-18T16:09:31.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.710+0000 7fbb1c9c5700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb18063d70 0x7fbb1814df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:31.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.710+0000 7fbb1c9c5700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbb180c6a60 0x7fbb18156660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:31.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.710+0000 7fbb1c9c5700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbb18153b50 0x7fbb18156b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:31.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.710+0000 7fbb1c9c5700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb180acaa0 con 0x7fbb18063d70 2023-10-18T16:09:31.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.710+0000 7fbb1c9c5700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb18154f00 con 0x7fbb18153b50 2023-10-18T16:09:31.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.710+0000 7fbb1c9c5700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb181561d0 con 0x7fbb180c6a60 2023-10-18T16:09:31.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.710+0000 7fbb177fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb18063d70 0x7fbb1814df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:31.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.710+0000 7fbb177fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb18063d70 0x7fbb1814df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41352/0 (socket says 172.21.15.88:41352) 2023-10-18T16:09:31.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.710+0000 7fbb177fe700 1 -- 172.21.15.88:0/1366226574 learned_addr learned my addr 172.21.15.88:0/1366226574 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:09:31.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.710+0000 7fbb16ffd700 1 --2- 172.21.15.88:0/1366226574 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbb180c6a60 0x7fbb18156660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:31.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.710+0000 7fbb17fff700 1 --2- 172.21.15.88:0/1366226574 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbb18153b50 0x7fbb18156b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:31.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.710+0000 7fbb177fe700 1 -- 172.21.15.88:0/1366226574 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbb180c6a60 msgr2=0x7fbb18156660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:31.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.710+0000 7fbb177fe700 1 --2- 172.21.15.88:0/1366226574 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbb180c6a60 0x7fbb18156660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:31.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.710+0000 7fbb177fe700 1 -- 172.21.15.88:0/1366226574 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbb18153b50 msgr2=0x7fbb18156b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:31.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.710+0000 7fbb177fe700 1 --2- 172.21.15.88:0/1366226574 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbb18153b50 0x7fbb18156b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:31.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.710+0000 7fbb177fe700 1 -- 172.21.15.88:0/1366226574 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb0c009370 con 0x7fbb18063d70 2023-10-18T16:09:31.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.710+0000 7fbb177fe700 1 --2- 172.21.15.88:0/1366226574 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb18063d70 0x7fbb1814df30 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7fbb08009f70 tx=0x7fbb180c3210).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:09:31.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.710+0000 7fbaff7fe700 1 -- 172.21.15.88:0/1366226574 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb0800bee0 con 0x7fbb18063d70 2023-10-18T16:09:31.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.710+0000 7fbaff7fe700 1 -- 172.21.15.88:0/1366226574 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb0801d420 con 0x7fbb18063d70 2023-10-18T16:09:31.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.710+0000 7fbb1c9c5700 1 -- 172.21.15.88:0/1366226574 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb08009a20 con 0x7fbb18063d70 2023-10-18T16:09:31.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.710+0000 7fbaff7fe700 1 -- 172.21.15.88:0/1366226574 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb0801d7a0 con 0x7fbb18063d70 2023-10-18T16:09:31.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.710+0000 7fbb1c9c5700 1 -- 172.21.15.88:0/1366226574 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb180bdba0 con 0x7fbb18063d70 2023-10-18T16:09:31.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.712+0000 7fbb1c9c5700 1 -- 172.21.15.88:0/1366226574 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb181564b0 con 0x7fbb18063d70 2023-10-18T16:09:31.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.712+0000 7fbaff7fe700 1 -- 172.21.15.88:0/1366226574 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7fbb08003040 con 0x7fbb18063d70 2023-10-18T16:09:31.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.712+0000 7fbaff7fe700 1 --2- 172.21.15.88:0/1366226574 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fbb0405f180 0x7fbb04061640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:31.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.712+0000 7fbaff7fe700 1 -- 172.21.15.88:0/1366226574 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6639+0+0 (secure 0 0 0) 0x7fbb0807bf20 con 0x7fbb18063d70 2023-10-18T16:09:31.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.712+0000 7fbb16ffd700 1 --2- 172.21.15.88:0/1366226574 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fbb0405f180 0x7fbb04061640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:31.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.713+0000 7fbb16ffd700 1 --2- 172.21.15.88:0/1366226574 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fbb0405f180 0x7fbb04061640 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fbb180c20a0 tx=0x7fbb0c0090f0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:09:31.728 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.716+0000 7fbaff7fe700 1 -- 172.21.15.88:0/1366226574 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7fbb0804cff0 con 0x7fbb18063d70 2023-10-18T16:09:31.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.922+0000 7fbb1c9c5700 1 -- 172.21.15.88:0/1366226574 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fbb1814c7e0 con 0x7fbb18063d70 2023-10-18T16:09:31.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.922+0000 7fbaff7fe700 1 -- 172.21.15.88:0/1366226574 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7fbb08050610 con 0x7fbb18063d70 2023-10-18T16:09:31.926 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:09:31.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.924+0000 7fbb1c9c5700 1 -- 172.21.15.88:0/1366226574 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fbb0405f180 msgr2=0x7fbb04061640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:31.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.924+0000 7fbb1c9c5700 1 --2- 172.21.15.88:0/1366226574 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fbb0405f180 0x7fbb04061640 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fbb180c20a0 tx=0x7fbb0c0090f0).stop 2023-10-18T16:09:31.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.924+0000 7fbb1c9c5700 1 -- 172.21.15.88:0/1366226574 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb18063d70 msgr2=0x7fbb1814df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:31.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.924+0000 7fbb1c9c5700 1 --2- 172.21.15.88:0/1366226574 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb18063d70 0x7fbb1814df30 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7fbb08009f70 tx=0x7fbb180c3210).stop 2023-10-18T16:09:31.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.924+0000 7fbb1c9c5700 1 -- 172.21.15.88:0/1366226574 shutdown_connections 2023-10-18T16:09:31.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.924+0000 7fbb1c9c5700 1 --2- 172.21.15.88:0/1366226574 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fbb0405f180 0x7fbb04061640 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:31.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.924+0000 7fbb1c9c5700 1 --2- 172.21.15.88:0/1366226574 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbb18153b50 0x7fbb18156b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:31.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.924+0000 7fbb1c9c5700 1 --2- 172.21.15.88:0/1366226574 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb18063d70 0x7fbb1814df30 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:31.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.924+0000 7fbb1c9c5700 1 --2- 172.21.15.88:0/1366226574 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbb180c6a60 0x7fbb18156660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:31.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.924+0000 7fbb1c9c5700 1 -- 172.21.15.88:0/1366226574 >> 172.21.15.88:0/1366226574 conn(0x7fbb1805e050 msgr2=0x7fbb180be650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:09:31.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.924+0000 7fbb1c9c5700 1 -- 172.21.15.88:0/1366226574 shutdown_connections 2023-10-18T16:09:31.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:31.924+0000 7fbb1c9c5700 1 -- 172.21.15.88:0/1366226574 wait complete. 2023-10-18T16:09:31.939 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:09:33.439 INFO:tasks.ceph.mds.a.smithi088.stderr:terminate called after throwing an instance of 'boost::wrapexcept' 2023-10-18T16:09:33.439 INFO:tasks.ceph.mds.a.smithi088.stderr: what(): boost::bad_get: failed value get using boost::get 2023-10-18T16:09:33.440 INFO:tasks.ceph.mds.a.smithi088.stderr:*** Caught signal (Aborted) ** 2023-10-18T16:09:33.440 INFO:tasks.ceph.mds.a.smithi088.stderr: in thread 7fb2c48cb700 thread_name:ms_dispatch 2023-10-18T16:09:33.440 INFO:tasks.ceph.mds.a.smithi088.stderr: ceph version 16.2.14-100-gb4ddd52c (b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5) pacific (stable) 2023-10-18T16:09:33.441 INFO:tasks.ceph.mds.a.smithi088.stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7fb2cc0ceb20] 2023-10-18T16:09:33.441 INFO:tasks.ceph.mds.a.smithi088.stderr: 2: gsignal() 2023-10-18T16:09:33.441 INFO:tasks.ceph.mds.a.smithi088.stderr: 3: abort() 2023-10-18T16:09:33.441 INFO:tasks.ceph.mds.a.smithi088.stderr: 4: /lib64/libstdc++.so.6(+0x9009b) [0x7fb2cb4cf09b] 2023-10-18T16:09:33.441 INFO:tasks.ceph.mds.a.smithi088.stderr: 5: /lib64/libstdc++.so.6(+0x9653c) [0x7fb2cb4d553c] 2023-10-18T16:09:33.442 INFO:tasks.ceph.mds.a.smithi088.stderr: 6: /lib64/libstdc++.so.6(+0x96597) [0x7fb2cb4d5597] 2023-10-18T16:09:33.442 INFO:tasks.ceph.mds.a.smithi088.stderr: 7: /lib64/libstdc++.so.6(+0x967f8) [0x7fb2cb4d57f8] 2023-10-18T16:09:33.442 INFO:tasks.ceph.mds.a.smithi088.stderr: 8: ceph-mds(+0x212b43) [0x565169777b43] 2023-10-18T16:09:33.442 INFO:tasks.ceph.mds.a.smithi088.stderr: 9: (Option::size_t const md_config_t::get_val(ConfigValues const&, std::basic_string_view >) const+0x5c) [0x5651697ec89c] 2023-10-18T16:09:33.442 INFO:tasks.ceph.mds.a.smithi088.stderr: 10: (SessionMap::SessionMap(MDSRank*)+0x2c2) [0x565169a346a2] 2023-10-18T16:09:33.443 INFO:tasks.ceph.mds.a.smithi088.stderr: 11: (MDSRank::MDSRank(int, std::__cxx11::basic_string, std::allocator >, ceph::fair_mutex&, std::shared_ptr&, CommonSafeTimer&, Beacon&, std::unique_ptr >&, Messenger*, MonClient*, MgrClient*, Context*, Context*, boost::asio::io_context&)+0x214) [0x565169751724] 2023-10-18T16:09:33.443 INFO:tasks.ceph.mds.a.smithi088.stderr: 12: (MDSRankDispatcher::MDSRankDispatcher(int, std::__cxx11::basic_string, std::allocator >, ceph::fair_mutex&, std::shared_ptr&, CommonSafeTimer&, Beacon&, std::unique_ptr >&, Messenger*, MonClient*, MgrClient*, Context*, Context*, boost::asio::io_context&)+0x101) [0x565169753181] 2023-10-18T16:09:33.443 INFO:tasks.ceph.mds.a.smithi088.stderr: 13: (MDSDaemon::handle_mds_map(boost::intrusive_ptr const&)+0x1608) [0x565169723348] 2023-10-18T16:09:33.443 INFO:tasks.ceph.mds.a.smithi088.stderr: 14: (MDSDaemon::handle_core_message(boost::intrusive_ptr const&)+0xcd) [0x56516972644d] 2023-10-18T16:09:33.443 INFO:tasks.ceph.mds.a.smithi088.stderr: 15: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0xc3) [0x565169726fa3] 2023-10-18T16:09:33.444 INFO:tasks.ceph.mds.a.smithi088.stderr: 16: (DispatchQueue::entry()+0x126a) [0x7fb2cd3284ea] 2023-10-18T16:09:33.444 INFO:tasks.ceph.mds.a.smithi088.stderr: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb2cd3dcd51] 2023-10-18T16:09:33.444 INFO:tasks.ceph.mds.a.smithi088.stderr: 18: /lib64/libpthread.so.0(+0x814a) [0x7fb2cc0c414a] 2023-10-18T16:09:33.444 INFO:tasks.ceph.mds.a.smithi088.stderr: 19: clone() 2023-10-18T16:09:33.444 INFO:tasks.ceph.mds.a.smithi088.stderr:2023-10-18T16:09:33.437+0000 7fb2c48cb700 -1 *** Caught signal (Aborted) ** 2023-10-18T16:09:33.444 INFO:tasks.ceph.mds.a.smithi088.stderr: in thread 7fb2c48cb700 thread_name:ms_dispatch 2023-10-18T16:09:33.445 INFO:tasks.ceph.mds.a.smithi088.stderr: 2023-10-18T16:09:33.445 INFO:tasks.ceph.mds.a.smithi088.stderr: ceph version 16.2.14-100-gb4ddd52c (b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5) pacific (stable) 2023-10-18T16:09:33.445 INFO:tasks.ceph.mds.a.smithi088.stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7fb2cc0ceb20] 2023-10-18T16:09:33.445 INFO:tasks.ceph.mds.a.smithi088.stderr: 2: gsignal() 2023-10-18T16:09:33.445 INFO:tasks.ceph.mds.a.smithi088.stderr: 3: abort() 2023-10-18T16:09:33.446 INFO:tasks.ceph.mds.a.smithi088.stderr: 4: /lib64/libstdc++.so.6(+0x9009b) [0x7fb2cb4cf09b] 2023-10-18T16:09:33.446 INFO:tasks.ceph.mds.a.smithi088.stderr: 5: /lib64/libstdc++.so.6(+0x9653c) [0x7fb2cb4d553c] 2023-10-18T16:09:33.446 INFO:tasks.ceph.mds.a.smithi088.stderr: 6: /lib64/libstdc++.so.6(+0x96597) [0x7fb2cb4d5597] 2023-10-18T16:09:33.446 INFO:tasks.ceph.mds.a.smithi088.stderr: 7: /lib64/libstdc++.so.6(+0x967f8) [0x7fb2cb4d57f8] 2023-10-18T16:09:33.446 INFO:tasks.ceph.mds.a.smithi088.stderr: 8: ceph-mds(+0x212b43) [0x565169777b43] 2023-10-18T16:09:33.447 INFO:tasks.ceph.mds.a.smithi088.stderr: 9: (Option::size_t const md_config_t::get_val(ConfigValues const&, std::basic_string_view >) const+0x5c) [0x5651697ec89c] 2023-10-18T16:09:33.447 INFO:tasks.ceph.mds.a.smithi088.stderr: 10: (SessionMap::SessionMap(MDSRank*)+0x2c2) [0x565169a346a2] 2023-10-18T16:09:33.447 INFO:tasks.ceph.mds.a.smithi088.stderr: 11: (MDSRank::MDSRank(int, std::__cxx11::basic_string, std::allocator >, ceph::fair_mutex&, std::shared_ptr&, CommonSafeTimer&, Beacon&, std::unique_ptr >&, Messenger*, MonClient*, MgrClient*, Context*, Context*, boost::asio::io_context&)+0x214) [0x565169751724] 2023-10-18T16:09:33.447 INFO:tasks.ceph.mds.a.smithi088.stderr: 12: (MDSRankDispatcher::MDSRankDispatcher(int, std::__cxx11::basic_string, std::allocator >, ceph::fair_mutex&, std::shared_ptr&, CommonSafeTimer&, Beacon&, std::unique_ptr >&, Messenger*, MonClient*, MgrClient*, Context*, Context*, boost::asio::io_context&)+0x101) [0x565169753181] 2023-10-18T16:09:33.447 INFO:tasks.ceph.mds.a.smithi088.stderr: 13: (MDSDaemon::handle_mds_map(boost::intrusive_ptr const&)+0x1608) [0x565169723348] 2023-10-18T16:09:33.448 INFO:tasks.ceph.mds.a.smithi088.stderr: 14: (MDSDaemon::handle_core_message(boost::intrusive_ptr const&)+0xcd) [0x56516972644d] 2023-10-18T16:09:33.448 INFO:tasks.ceph.mds.a.smithi088.stderr: 15: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0xc3) [0x565169726fa3] 2023-10-18T16:09:33.448 INFO:tasks.ceph.mds.a.smithi088.stderr: 16: (DispatchQueue::entry()+0x126a) [0x7fb2cd3284ea] 2023-10-18T16:09:33.448 INFO:tasks.ceph.mds.a.smithi088.stderr: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb2cd3dcd51] 2023-10-18T16:09:33.448 INFO:tasks.ceph.mds.a.smithi088.stderr: 18: /lib64/libpthread.so.0(+0x814a) [0x7fb2cc0c414a] 2023-10-18T16:09:33.448 INFO:tasks.ceph.mds.a.smithi088.stderr: 19: clone() 2023-10-18T16:09:33.449 INFO:tasks.ceph.mds.a.smithi088.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-10-18T16:09:33.449 INFO:tasks.ceph.mds.a.smithi088.stderr: 2023-10-18T16:09:33.449 INFO:tasks.ceph.mds.a.smithi088.stderr: 0> 2023-10-18T16:09:33.437+0000 7fb2c48cb700 -1 *** Caught signal (Aborted) ** 2023-10-18T16:09:33.450 INFO:tasks.ceph.mds.a.smithi088.stderr: in thread 7fb2c48cb700 thread_name:ms_dispatch 2023-10-18T16:09:33.450 INFO:tasks.ceph.mds.a.smithi088.stderr: 2023-10-18T16:09:33.450 INFO:tasks.ceph.mds.a.smithi088.stderr: ceph version 16.2.14-100-gb4ddd52c (b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5) pacific (stable) 2023-10-18T16:09:33.450 INFO:tasks.ceph.mds.a.smithi088.stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7fb2cc0ceb20] 2023-10-18T16:09:33.450 INFO:tasks.ceph.mds.a.smithi088.stderr: 2: gsignal() 2023-10-18T16:09:33.451 INFO:tasks.ceph.mds.a.smithi088.stderr: 3: abort() 2023-10-18T16:09:33.451 INFO:tasks.ceph.mds.a.smithi088.stderr: 4: /lib64/libstdc++.so.6(+0x9009b) [0x7fb2cb4cf09b] 2023-10-18T16:09:33.451 INFO:tasks.ceph.mds.a.smithi088.stderr: 5: /lib64/libstdc++.so.6(+0x9653c) [0x7fb2cb4d553c] 2023-10-18T16:09:33.451 INFO:tasks.ceph.mds.a.smithi088.stderr: 6: /lib64/libstdc++.so.6(+0x96597) [0x7fb2cb4d5597] 2023-10-18T16:09:33.451 INFO:tasks.ceph.mds.a.smithi088.stderr: 7: /lib64/libstdc++.so.6(+0x967f8) [0x7fb2cb4d57f8] 2023-10-18T16:09:33.452 INFO:tasks.ceph.mds.a.smithi088.stderr: 8: ceph-mds(+0x212b43) [0x565169777b43] 2023-10-18T16:09:33.452 INFO:tasks.ceph.mds.a.smithi088.stderr: 9: (Option::size_t const md_config_t::get_val(ConfigValues const&, std::basic_string_view >) const+0x5c) [0x5651697ec89c] 2023-10-18T16:09:33.452 INFO:tasks.ceph.mds.a.smithi088.stderr: 10: (SessionMap::SessionMap(MDSRank*)+0x2c2) [0x565169a346a2] 2023-10-18T16:09:33.452 INFO:tasks.ceph.mds.a.smithi088.stderr: 11: (MDSRank::MDSRank(int, std::__cxx11::basic_string, std::allocator >, ceph::fair_mutex&, std::shared_ptr&, CommonSafeTimer&, Beacon&, std::unique_ptr >&, Messenger*, MonClient*, MgrClient*, Context*, Context*, boost::asio::io_context&)+0x214) [0x565169751724] 2023-10-18T16:09:33.452 INFO:tasks.ceph.mds.a.smithi088.stderr: 12: (MDSRankDispatcher::MDSRankDispatcher(int, std::__cxx11::basic_string, std::allocator >, ceph::fair_mutex&, std::shared_ptr&, CommonSafeTimer&, Beacon&, std::unique_ptr >&, Messenger*, MonClient*, MgrClient*, Context*, Context*, boost::asio::io_context&)+0x101) [0x565169753181] 2023-10-18T16:09:33.452 INFO:tasks.ceph.mds.a.smithi088.stderr: 13: (MDSDaemon::handle_mds_map(boost::intrusive_ptr const&)+0x1608) [0x565169723348] 2023-10-18T16:09:33.453 INFO:tasks.ceph.mds.a.smithi088.stderr: 14: (MDSDaemon::handle_core_message(boost::intrusive_ptr const&)+0xcd) [0x56516972644d] 2023-10-18T16:09:33.453 INFO:tasks.ceph.mds.a.smithi088.stderr: 15: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0xc3) [0x565169726fa3] 2023-10-18T16:09:33.453 INFO:tasks.ceph.mds.a.smithi088.stderr: 16: (DispatchQueue::entry()+0x126a) [0x7fb2cd3284ea] 2023-10-18T16:09:33.453 INFO:tasks.ceph.mds.a.smithi088.stderr: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb2cd3dcd51] 2023-10-18T16:09:33.453 INFO:tasks.ceph.mds.a.smithi088.stderr: 18: /lib64/libpthread.so.0(+0x814a) [0x7fb2cc0c414a] 2023-10-18T16:09:33.454 INFO:tasks.ceph.mds.a.smithi088.stderr: 19: clone() 2023-10-18T16:09:33.454 INFO:tasks.ceph.mds.a.smithi088.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-10-18T16:09:33.454 INFO:tasks.ceph.mds.a.smithi088.stderr: 2023-10-18T16:09:33.454 INFO:tasks.ceph.mds.a.smithi088.stderr: 0> 2023-10-18T16:09:33.437+0000 7fb2c48cb700 -1 *** Caught signal (Aborted) ** 2023-10-18T16:09:33.454 INFO:tasks.ceph.mds.a.smithi088.stderr: in thread 7fb2c48cb700 thread_name:ms_dispatch 2023-10-18T16:09:33.455 INFO:tasks.ceph.mds.a.smithi088.stderr: 2023-10-18T16:09:33.455 INFO:tasks.ceph.mds.a.smithi088.stderr: ceph version 16.2.14-100-gb4ddd52c (b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5) pacific (stable) 2023-10-18T16:09:33.455 INFO:tasks.ceph.mds.a.smithi088.stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7fb2cc0ceb20] 2023-10-18T16:09:33.455 INFO:tasks.ceph.mds.a.smithi088.stderr: 2: gsignal() 2023-10-18T16:09:33.455 INFO:tasks.ceph.mds.a.smithi088.stderr: 3: abort() 2023-10-18T16:09:33.456 INFO:tasks.ceph.mds.a.smithi088.stderr: 4: /lib64/libstdc++.so.6(+0x9009b) [0x7fb2cb4cf09b] 2023-10-18T16:09:33.456 INFO:tasks.ceph.mds.a.smithi088.stderr: 5: /lib64/libstdc++.so.6(+0x9653c) [0x7fb2cb4d553c] 2023-10-18T16:09:33.456 INFO:tasks.ceph.mds.a.smithi088.stderr: 6: /lib64/libstdc++.so.6(+0x96597) [0x7fb2cb4d5597] 2023-10-18T16:09:33.456 INFO:tasks.ceph.mds.a.smithi088.stderr: 7: /lib64/libstdc++.so.6(+0x967f8) [0x7fb2cb4d57f8] 2023-10-18T16:09:33.456 INFO:tasks.ceph.mds.a.smithi088.stderr: 8: ceph-mds(+0x212b43) [0x565169777b43] 2023-10-18T16:09:33.456 INFO:tasks.ceph.mds.a.smithi088.stderr: 9: (Option::size_t const md_config_t::get_val(ConfigValues const&, std::basic_string_view >) const+0x5c) [0x5651697ec89c] 2023-10-18T16:09:33.457 INFO:tasks.ceph.mds.a.smithi088.stderr: 10: (SessionMap::SessionMap(MDSRank*)+0x2c2) [0x565169a346a2] 2023-10-18T16:09:33.457 INFO:tasks.ceph.mds.a.smithi088.stderr: 11: (MDSRank::MDSRank(int, std::__cxx11::basic_string, std::allocator >, ceph::fair_mutex&, std::shared_ptr&, CommonSafeTimer&, Beacon&, std::unique_ptr >&, Messenger*, MonClient*, MgrClient*, Context*, Context*, boost::asio::io_context&)+0x214) [0x565169751724] 2023-10-18T16:09:33.457 INFO:tasks.ceph.mds.a.smithi088.stderr: 12: (MDSRankDispatcher::MDSRankDispatcher(int, std::__cxx11::basic_string, std::allocator >, ceph::fair_mutex&, std::shared_ptr&, CommonSafeTimer&, Beacon&, std::unique_ptr >&, Messenger*, MonClient*, MgrClient*, Context*, Context*, boost::asio::io_context&)+0x101) [0x565169753181] 2023-10-18T16:09:33.457 INFO:tasks.ceph.mds.a.smithi088.stderr: 13: (MDSDaemon::handle_mds_map(boost::intrusive_ptr const&)+0x1608) [0x565169723348] 2023-10-18T16:09:33.457 INFO:tasks.ceph.mds.a.smithi088.stderr: 14: (MDSDaemon::handle_core_message(boost::intrusive_ptr const&)+0xcd) [0x56516972644d] 2023-10-18T16:09:33.458 INFO:tasks.ceph.mds.a.smithi088.stderr: 15: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0xc3) [0x565169726fa3] 2023-10-18T16:09:33.458 INFO:tasks.ceph.mds.a.smithi088.stderr: 16: (DispatchQueue::entry()+0x126a) [0x7fb2cd3284ea] 2023-10-18T16:09:33.458 INFO:tasks.ceph.mds.a.smithi088.stderr: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb2cd3dcd51] 2023-10-18T16:09:33.458 INFO:tasks.ceph.mds.a.smithi088.stderr: 18: /lib64/libpthread.so.0(+0x814a) [0x7fb2cc0c414a] 2023-10-18T16:09:33.458 INFO:tasks.ceph.mds.a.smithi088.stderr: 19: clone() 2023-10-18T16:09:33.459 INFO:tasks.ceph.mds.a.smithi088.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-10-18T16:09:33.459 INFO:tasks.ceph.mds.a.smithi088.stderr: 2023-10-18T16:09:33.573 INFO:tasks.ceph.mds.a.smithi088.stderr:daemon-helper: command crashed with signal 6 2023-10-18T16:09:34.940 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:09:35.063 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.060+0000 7f3c97971700 1 Processor -- start 2023-10-18T16:09:35.064 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.060+0000 7f3c97971700 1 -- start start 2023-10-18T16:09:35.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.061+0000 7f3c97971700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3c90153aa0 0x7f3c90153e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:35.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.061+0000 7f3c97971700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3c900654b0 0x7f3c90154370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:35.067 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.061+0000 7f3c97971700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3c900bf230 0x7f3c90066b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:35.067 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.061+0000 7f3c97971700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c90069680 con 0x7f3c90153aa0 2023-10-18T16:09:35.067 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.061+0000 7f3c97971700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c90063c70 con 0x7f3c900654b0 2023-10-18T16:09:35.067 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.061+0000 7f3c97971700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c900bf700 con 0x7f3c900bf230 2023-10-18T16:09:35.067 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.061+0000 7f3c9570d700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3c90153aa0 0x7f3c90153e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:35.068 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.061+0000 7f3c9570d700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3c90153aa0 0x7f3c90153e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41362/0 (socket says 172.21.15.88:41362) 2023-10-18T16:09:35.068 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.061+0000 7f3c9570d700 1 -- 172.21.15.88:0/2499102767 learned_addr learned my addr 172.21.15.88:0/2499102767 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:09:35.068 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.061+0000 7f3c9570d700 1 -- 172.21.15.88:0/2499102767 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3c900bf230 msgr2=0x7f3c90066b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:09:35.068 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.061+0000 7f3c95f0e700 1 --2- 172.21.15.88:0/2499102767 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3c900bf230 0x7f3c90066b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:35.068 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.061+0000 7f3c94f0c700 1 --2- 172.21.15.88:0/2499102767 >> v2:172.21.15.151:3300/0 conn(0x7f3c900654b0 0x7f3c90154370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:35.068 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.061+0000 7f3c9570d700 1 --2- 172.21.15.88:0/2499102767 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3c900bf230 0x7f3c90066b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:35.069 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.061+0000 7f3c9570d700 1 -- 172.21.15.88:0/2499102767 >> v2:172.21.15.151:3300/0 conn(0x7f3c900654b0 msgr2=0x7f3c90154370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:09:35.069 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.062+0000 7f3c9570d700 1 --2- 172.21.15.88:0/2499102767 >> v2:172.21.15.151:3300/0 conn(0x7f3c900654b0 0x7f3c90154370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:35.069 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.062+0000 7f3c9570d700 1 -- 172.21.15.88:0/2499102767 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c900690c0 con 0x7f3c90153aa0 2023-10-18T16:09:35.069 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.062+0000 7f3c94f0c700 1 --2- 172.21.15.88:0/2499102767 >> v2:172.21.15.151:3300/0 conn(0x7f3c900654b0 0x7f3c90154370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-10-18T16:09:35.069 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.062+0000 7f3c9570d700 1 --2- 172.21.15.88:0/2499102767 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c90153aa0 0x7f3c90153e80 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f3c9006b210 tx=0x7f3c8c0090f0).ready entity=mon.0 client_cookie=58481d57ebdde497 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:09:35.070 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.062+0000 7f3c87fff700 1 -- 172.21.15.88:0/2499102767 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c8c00b690 con 0x7f3c90153aa0 2023-10-18T16:09:35.070 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.062+0000 7f3c87fff700 1 -- 172.21.15.88:0/2499102767 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c8c00b7f0 con 0x7f3c90153aa0 2023-10-18T16:09:35.070 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.062+0000 7f3c87fff700 1 -- 172.21.15.88:0/2499102767 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c8c00bb70 con 0x7f3c90153aa0 2023-10-18T16:09:35.070 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.062+0000 7f3c97971700 1 -- 172.21.15.88:0/2499102767 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c90153aa0 msgr2=0x7f3c90153e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:35.070 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.062+0000 7f3c97971700 1 --2- 172.21.15.88:0/2499102767 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c90153aa0 0x7f3c90153e80 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f3c9006b210 tx=0x7f3c8c0090f0).stop 2023-10-18T16:09:35.071 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.062+0000 7f3c97971700 1 -- 172.21.15.88:0/2499102767 shutdown_connections 2023-10-18T16:09:35.071 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.062+0000 7f3c97971700 1 --2- 172.21.15.88:0/2499102767 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3c900bf230 0x7f3c90066b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:35.071 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.062+0000 7f3c97971700 1 --2- 172.21.15.88:0/2499102767 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c90153aa0 0x7f3c90153e80 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:35.071 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.062+0000 7f3c97971700 1 --2- 172.21.15.88:0/2499102767 >> v2:172.21.15.151:3300/0 conn(0x7f3c900654b0 0x7f3c90154370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:35.071 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.062+0000 7f3c97971700 1 -- 172.21.15.88:0/2499102767 >> 172.21.15.88:0/2499102767 conn(0x7f3c900c4890 msgr2=0x7f3c900c4ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:09:35.071 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.062+0000 7f3c97971700 1 -- 172.21.15.88:0/2499102767 shutdown_connections 2023-10-18T16:09:35.072 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.062+0000 7f3c97971700 1 -- 172.21.15.88:0/2499102767 wait complete. 2023-10-18T16:09:35.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.063+0000 7f3c97971700 1 Processor -- start 2023-10-18T16:09:35.075 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.064+0000 7f3c97971700 1 -- start start 2023-10-18T16:09:35.075 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.064+0000 7f3c97971700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c900654b0 0x7f3c9014dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:35.075 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.064+0000 7f3c97971700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3c900bf230 0x7f3c900bc2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:35.075 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.064+0000 7f3c97971700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3c90153aa0 0x7f3c9014bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:35.075 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.064+0000 7f3c97971700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c90163f40 con 0x7f3c900654b0 2023-10-18T16:09:35.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.064+0000 7f3c97971700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c900bce40 con 0x7f3c900bf230 2023-10-18T16:09:35.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.064+0000 7f3c97971700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c900c1b80 con 0x7f3c90153aa0 2023-10-18T16:09:35.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.065+0000 7f3c9570d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c900654b0 0x7f3c9014dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:35.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.065+0000 7f3c9570d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c900654b0 0x7f3c9014dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41374/0 (socket says 172.21.15.88:41374) 2023-10-18T16:09:35.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.065+0000 7f3c9570d700 1 -- 172.21.15.88:0/4131548346 learned_addr learned my addr 172.21.15.88:0/4131548346 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:09:35.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.065+0000 7f3c94f0c700 1 --2- 172.21.15.88:0/4131548346 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3c900bf230 0x7f3c900bc2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:35.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.065+0000 7f3c95f0e700 1 --2- 172.21.15.88:0/4131548346 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3c90153aa0 0x7f3c9014bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:35.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.065+0000 7f3c9570d700 1 -- 172.21.15.88:0/4131548346 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3c90153aa0 msgr2=0x7f3c9014bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:35.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.065+0000 7f3c9570d700 1 --2- 172.21.15.88:0/4131548346 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3c90153aa0 0x7f3c9014bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:35.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.065+0000 7f3c9570d700 1 -- 172.21.15.88:0/4131548346 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3c900bf230 msgr2=0x7f3c900bc2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:35.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.065+0000 7f3c9570d700 1 --2- 172.21.15.88:0/4131548346 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3c900bf230 0x7f3c900bc2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:35.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.065+0000 7f3c9570d700 1 -- 172.21.15.88:0/4131548346 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c8c009370 con 0x7f3c900654b0 2023-10-18T16:09:35.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.065+0000 7f3c95f0e700 1 --2- 172.21.15.88:0/4131548346 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3c90153aa0 0x7f3c9014bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-10-18T16:09:35.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.065+0000 7f3c9570d700 1 --2- 172.21.15.88:0/4131548346 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c900654b0 0x7f3c9014dc20 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f3c8c009130 tx=0x7f3c8c017000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:09:35.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.065+0000 7f3c857fa700 1 -- 172.21.15.88:0/4131548346 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c8c00bb70 con 0x7f3c900654b0 2023-10-18T16:09:35.079 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.065+0000 7f3c857fa700 1 -- 172.21.15.88:0/4131548346 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c8c000d10 con 0x7f3c900654b0 2023-10-18T16:09:35.079 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.065+0000 7f3c97971700 1 -- 172.21.15.88:0/4131548346 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c900bc920 con 0x7f3c900654b0 2023-10-18T16:09:35.079 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.065+0000 7f3c857fa700 1 -- 172.21.15.88:0/4131548346 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c8c01b660 con 0x7f3c900654b0 2023-10-18T16:09:35.079 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.065+0000 7f3c97971700 1 -- 172.21.15.88:0/4131548346 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c901638f0 con 0x7f3c900654b0 2023-10-18T16:09:35.079 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.066+0000 7f3c97971700 1 -- 172.21.15.88:0/4131548346 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c90164220 con 0x7f3c900654b0 2023-10-18T16:09:35.079 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.066+0000 7f3c857fa700 1 -- 172.21.15.88:0/4131548346 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f3c8c01ba10 con 0x7f3c900654b0 2023-10-18T16:09:35.080 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.067+0000 7f3c857fa700 1 --2- 172.21.15.88:0/4131548346 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f3c7405f150 0x7f3c74061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:35.080 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.067+0000 7f3c857fa700 1 -- 172.21.15.88:0/4131548346 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6725+0+0 (secure 0 0 0) 0x7f3c8c07b370 con 0x7f3c900654b0 2023-10-18T16:09:35.080 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.067+0000 7f3c94f0c700 1 --2- 172.21.15.88:0/4131548346 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f3c7405f150 0x7f3c74061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:35.080 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.067+0000 7f3c94f0c700 1 --2- 172.21.15.88:0/4131548346 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f3c7405f150 0x7f3c74061610 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f3c900bf110 tx=0x7f3c80008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:09:35.080 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.071+0000 7f3c857fa700 1 -- 172.21.15.88:0/4131548346 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f3c8c04c490 con 0x7f3c900654b0 2023-10-18T16:09:35.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.270+0000 7f3c97971700 1 -- 172.21.15.88:0/4131548346 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3c9006c4e0 con 0x7f3c900654b0 2023-10-18T16:09:35.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.271+0000 7f3c857fa700 1 -- 172.21.15.88:0/4131548346 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7f3c8c021080 con 0x7f3c900654b0 2023-10-18T16:09:35.275 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:09:35.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.273+0000 7f3c97971700 1 -- 172.21.15.88:0/4131548346 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f3c7405f150 msgr2=0x7f3c74061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:35.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.273+0000 7f3c97971700 1 --2- 172.21.15.88:0/4131548346 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f3c7405f150 0x7f3c74061610 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f3c900bf110 tx=0x7f3c80008040).stop 2023-10-18T16:09:35.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.273+0000 7f3c97971700 1 -- 172.21.15.88:0/4131548346 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c900654b0 msgr2=0x7f3c9014dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:35.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.273+0000 7f3c97971700 1 --2- 172.21.15.88:0/4131548346 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c900654b0 0x7f3c9014dc20 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f3c8c009130 tx=0x7f3c8c017000).stop 2023-10-18T16:09:35.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.273+0000 7f3c97971700 1 -- 172.21.15.88:0/4131548346 shutdown_connections 2023-10-18T16:09:35.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.273+0000 7f3c97971700 1 --2- 172.21.15.88:0/4131548346 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f3c7405f150 0x7f3c74061610 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:35.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.273+0000 7f3c97971700 1 --2- 172.21.15.88:0/4131548346 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3c90153aa0 0x7f3c9014bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:35.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.273+0000 7f3c97971700 1 --2- 172.21.15.88:0/4131548346 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c900654b0 0x7f3c9014dc20 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:35.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.273+0000 7f3c97971700 1 --2- 172.21.15.88:0/4131548346 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3c900bf230 0x7f3c900bc2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:35.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.273+0000 7f3c97971700 1 -- 172.21.15.88:0/4131548346 >> 172.21.15.88:0/4131548346 conn(0x7f3c900c4890 msgr2=0x7f3c900c3660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:09:35.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.273+0000 7f3c97971700 1 -- 172.21.15.88:0/4131548346 shutdown_connections 2023-10-18T16:09:35.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:35.273+0000 7f3c97971700 1 -- 172.21.15.88:0/4131548346 wait complete. 2023-10-18T16:09:35.288 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:09:35.885 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~0s 2023-10-18T16:09:35.886 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~13s 2023-10-18T16:09:35.886 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~46s 2023-10-18T16:09:35.886 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~26s 2023-10-18T16:09:35.886 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~107s 2023-10-18T16:09:35.887 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.f is failed for ~59s 2023-10-18T16:09:38.289 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:09:38.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.409+0000 7f0bb4738700 1 Processor -- start 2023-10-18T16:09:38.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.409+0000 7f0bb4738700 1 -- start start 2023-10-18T16:09:38.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.410+0000 7f0bb4738700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f0bac0bf4a0 0x7f0bac0bf880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:38.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.410+0000 7f0bb4738700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0bac067e50 0x7f0bac0672b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:38.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.410+0000 7f0bb4738700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bac0654b0 0x7f0bac1538c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:38.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.410+0000 7f0bb4738700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bac066e20 con 0x7f0bac067e50 2023-10-18T16:09:38.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.410+0000 7f0bb4738700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bac151210 con 0x7f0bac0bf4a0 2023-10-18T16:09:38.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.410+0000 7f0bb4738700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bac068650 con 0x7f0bac0654b0 2023-10-18T16:09:38.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.411+0000 7f0bb1cd3700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0bac067e50 0x7f0bac0672b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:38.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.411+0000 7f0bb1cd3700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0bac067e50 0x7f0bac0672b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37182/0 (socket says 172.21.15.88:37182) 2023-10-18T16:09:38.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.411+0000 7f0bb1cd3700 1 -- 172.21.15.88:0/3000729874 learned_addr learned my addr 172.21.15.88:0/3000729874 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:09:38.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.411+0000 7f0bb24d4700 1 --2- 172.21.15.88:0/3000729874 >> v2:172.21.15.151:3300/0 conn(0x7f0bac0bf4a0 0x7f0bac0bf880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:38.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.411+0000 7f0bb2cd5700 1 --2- 172.21.15.88:0/3000729874 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bac0654b0 0x7f0bac1538c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:38.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.411+0000 7f0bb1cd3700 1 -- 172.21.15.88:0/3000729874 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bac0654b0 msgr2=0x7f0bac1538c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:09:38.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.411+0000 7f0bb1cd3700 1 --2- 172.21.15.88:0/3000729874 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bac0654b0 0x7f0bac1538c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:38.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.411+0000 7f0bb1cd3700 1 -- 172.21.15.88:0/3000729874 >> v2:172.21.15.151:3300/0 conn(0x7f0bac0bf4a0 msgr2=0x7f0bac0bf880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:09:38.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.411+0000 7f0bb1cd3700 1 --2- 172.21.15.88:0/3000729874 >> v2:172.21.15.151:3300/0 conn(0x7f0bac0bf4a0 0x7f0bac0bf880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:38.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.411+0000 7f0bb1cd3700 1 -- 172.21.15.88:0/3000729874 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bac063800 con 0x7f0bac067e50 2023-10-18T16:09:38.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.411+0000 7f0bb1cd3700 1 --2- 172.21.15.88:0/3000729874 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0bac067e50 0x7f0bac0672b0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f0bac0c55f0 tx=0x7f0b9c0090f0).ready entity=mon.0 client_cookie=185a68494d801c61 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:09:38.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.411+0000 7f0bb14d2700 1 -- 172.21.15.88:0/3000729874 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b9c00b600 con 0x7f0bac067e50 2023-10-18T16:09:38.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.411+0000 7f0bb14d2700 1 -- 172.21.15.88:0/3000729874 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b9c00b760 con 0x7f0bac067e50 2023-10-18T16:09:38.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.411+0000 7f0bb14d2700 1 -- 172.21.15.88:0/3000729874 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b9c00bae0 con 0x7f0bac067e50 2023-10-18T16:09:38.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.412+0000 7f0bb4738700 1 -- 172.21.15.88:0/3000729874 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0bac067e50 msgr2=0x7f0bac0672b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:38.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.412+0000 7f0bb4738700 1 --2- 172.21.15.88:0/3000729874 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0bac067e50 0x7f0bac0672b0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f0bac0c55f0 tx=0x7f0b9c0090f0).stop 2023-10-18T16:09:38.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.412+0000 7f0bb4738700 1 -- 172.21.15.88:0/3000729874 shutdown_connections 2023-10-18T16:09:38.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.412+0000 7f0bb4738700 1 --2- 172.21.15.88:0/3000729874 >> v2:172.21.15.151:3300/0 conn(0x7f0bac0bf4a0 0x7f0bac0bf880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:38.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.412+0000 7f0bb4738700 1 --2- 172.21.15.88:0/3000729874 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bac0654b0 0x7f0bac1538c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:38.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.412+0000 7f0bb4738700 1 --2- 172.21.15.88:0/3000729874 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0bac067e50 0x7f0bac0672b0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:38.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.412+0000 7f0bb4738700 1 -- 172.21.15.88:0/3000729874 >> 172.21.15.88:0/3000729874 conn(0x7f0bac0c4940 msgr2=0x7f0bac0c4d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:09:38.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.412+0000 7f0bb4738700 1 -- 172.21.15.88:0/3000729874 shutdown_connections 2023-10-18T16:09:38.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.412+0000 7f0bb4738700 1 -- 172.21.15.88:0/3000729874 wait complete. 2023-10-18T16:09:38.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.413+0000 7f0bb4738700 1 Processor -- start 2023-10-18T16:09:38.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.413+0000 7f0bb4738700 1 -- start start 2023-10-18T16:09:38.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.414+0000 7f0bb4738700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0bac0654b0 0x7f0bac14a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:38.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.414+0000 7f0bb4738700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bac067e50 0x7f0bac160220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:38.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.414+0000 7f0bb4738700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0bac0bf4a0 0x7f0bac160710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:38.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.414+0000 7f0bb4738700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bac15fdd0 con 0x7f0bac0bf4a0 2023-10-18T16:09:38.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.414+0000 7f0bb4738700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bac154700 con 0x7f0bac0654b0 2023-10-18T16:09:38.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.414+0000 7f0bb4738700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bac160c00 con 0x7f0bac067e50 2023-10-18T16:09:38.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.414+0000 7f0bb2cd5700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0bac0bf4a0 0x7f0bac160710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:38.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.414+0000 7f0bb2cd5700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0bac0bf4a0 0x7f0bac160710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37198/0 (socket says 172.21.15.88:37198) 2023-10-18T16:09:38.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.414+0000 7f0bb2cd5700 1 -- 172.21.15.88:0/2093354626 learned_addr learned my addr 172.21.15.88:0/2093354626 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:09:38.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.414+0000 7f0bb1cd3700 1 --2- 172.21.15.88:0/2093354626 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bac067e50 0x7f0bac160220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:38.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.414+0000 7f0bb24d4700 1 --2- 172.21.15.88:0/2093354626 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0bac0654b0 0x7f0bac14a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:38.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.414+0000 7f0bb2cd5700 1 -- 172.21.15.88:0/2093354626 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bac067e50 msgr2=0x7f0bac160220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:38.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.414+0000 7f0bb2cd5700 1 --2- 172.21.15.88:0/2093354626 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bac067e50 0x7f0bac160220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:38.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.414+0000 7f0bb2cd5700 1 -- 172.21.15.88:0/2093354626 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0bac0654b0 msgr2=0x7f0bac14a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:38.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.414+0000 7f0bb2cd5700 1 --2- 172.21.15.88:0/2093354626 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0bac0654b0 0x7f0bac14a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:38.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.414+0000 7f0bb2cd5700 1 -- 172.21.15.88:0/2093354626 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b9c009370 con 0x7f0bac0bf4a0 2023-10-18T16:09:38.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.414+0000 7f0bb2cd5700 1 --2- 172.21.15.88:0/2093354626 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0bac0bf4a0 0x7f0bac160710 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f0ba400d890 tx=0x7f0ba4008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:09:38.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.414+0000 7f0ba27fc700 1 -- 172.21.15.88:0/2093354626 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ba4011590 con 0x7f0bac0bf4a0 2023-10-18T16:09:38.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.414+0000 7f0ba27fc700 1 -- 172.21.15.88:0/2093354626 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ba40116f0 con 0x7f0bac0bf4a0 2023-10-18T16:09:38.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.414+0000 7f0bb4738700 1 -- 172.21.15.88:0/2093354626 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bac14f170 con 0x7f0bac0bf4a0 2023-10-18T16:09:38.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.414+0000 7f0ba27fc700 1 -- 172.21.15.88:0/2093354626 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ba4011a70 con 0x7f0bac0bf4a0 2023-10-18T16:09:38.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.414+0000 7f0bb4738700 1 -- 172.21.15.88:0/2093354626 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bac14f410 con 0x7f0bac0bf4a0 2023-10-18T16:09:38.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.415+0000 7f0bb4738700 1 -- 172.21.15.88:0/2093354626 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bac15ff40 con 0x7f0bac0bf4a0 2023-10-18T16:09:38.430 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.416+0000 7f0ba27fc700 1 -- 172.21.15.88:0/2093354626 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f0ba4012420 con 0x7f0bac0bf4a0 2023-10-18T16:09:38.430 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.416+0000 7f0ba27fc700 1 --2- 172.21.15.88:0/2093354626 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f0b9005f150 0x7f0b90061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:38.430 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.416+0000 7f0ba27fc700 1 -- 172.21.15.88:0/2093354626 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6725+0+0 (secure 0 0 0) 0x7f0ba4079d50 con 0x7f0bac0bf4a0 2023-10-18T16:09:38.430 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.416+0000 7f0bb24d4700 1 --2- 172.21.15.88:0/2093354626 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f0b9005f150 0x7f0b90061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:38.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.417+0000 7f0bb24d4700 1 --2- 172.21.15.88:0/2093354626 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f0b9005f150 0x7f0b90061610 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f0ba80075a0 tx=0x7f0ba80097e0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:09:38.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.420+0000 7f0ba27fc700 1 -- 172.21.15.88:0/2093354626 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f0bac15ff40 con 0x7f0bac0bf4a0 2023-10-18T16:09:38.431 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-18T16:09:38.433 DEBUG:teuthology.orchestra.run.smithi151:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-18T16:09:38.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.627+0000 7f0bb4738700 1 -- 172.21.15.88:0/2093354626 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f0bac15ff10 con 0x7f0bac0bf4a0 2023-10-18T16:09:38.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.627+0000 7f0ba27fc700 1 -- 172.21.15.88:0/2093354626 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7f0ba407d030 con 0x7f0bac0bf4a0 2023-10-18T16:09:38.631 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:09:38.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.628+0000 7f0bb4738700 1 -- 172.21.15.88:0/2093354626 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f0b9005f150 msgr2=0x7f0b90061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:38.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.628+0000 7f0bb4738700 1 --2- 172.21.15.88:0/2093354626 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f0b9005f150 0x7f0b90061610 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f0ba80075a0 tx=0x7f0ba80097e0).stop 2023-10-18T16:09:38.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.628+0000 7f0bb4738700 1 -- 172.21.15.88:0/2093354626 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0bac0bf4a0 msgr2=0x7f0bac160710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:38.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.628+0000 7f0bb4738700 1 --2- 172.21.15.88:0/2093354626 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0bac0bf4a0 0x7f0bac160710 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f0ba400d890 tx=0x7f0ba4008040).stop 2023-10-18T16:09:38.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.628+0000 7f0bb4738700 1 -- 172.21.15.88:0/2093354626 shutdown_connections 2023-10-18T16:09:38.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.629+0000 7f0bb4738700 1 --2- 172.21.15.88:0/2093354626 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f0b9005f150 0x7f0b90061610 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:38.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.629+0000 7f0bb4738700 1 --2- 172.21.15.88:0/2093354626 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0bac0bf4a0 0x7f0bac160710 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:38.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.629+0000 7f0bb4738700 1 --2- 172.21.15.88:0/2093354626 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0bac0654b0 0x7f0bac14a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:38.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.629+0000 7f0bb4738700 1 --2- 172.21.15.88:0/2093354626 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bac067e50 0x7f0bac160220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:38.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.629+0000 7f0bb4738700 1 -- 172.21.15.88:0/2093354626 >> 172.21.15.88:0/2093354626 conn(0x7f0bac0c4940 msgr2=0x7f0bac14c400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:09:38.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.629+0000 7f0bb4738700 1 -- 172.21.15.88:0/2093354626 shutdown_connections 2023-10-18T16:09:38.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:38.629+0000 7f0bb4738700 1 -- 172.21.15.88:0/2093354626 wait complete. 2023-10-18T16:09:38.644 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:09:41.645 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:09:41.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.766+0000 7f7945e06700 1 Processor -- start 2023-10-18T16:09:41.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.766+0000 7f7945e06700 1 -- start start 2023-10-18T16:09:41.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.767+0000 7f7945e06700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f794014d8c0 0x7f7940064440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:41.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.767+0000 7f7945e06700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f79400a9050 0x7f7940064930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:41.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.767+0000 7f7945e06700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7940064150 con 0x7f7940065360 2023-10-18T16:09:41.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.767+0000 7f7945e06700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7940153a40 con 0x7f794014d8c0 2023-10-18T16:09:41.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.767+0000 7f7945e06700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7940063e70 con 0x7f79400a9050 2023-10-18T16:09:41.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.767+0000 7f793f7fe700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f794014d8c0 0x7f7940064440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:41.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.767+0000 7f793f7fe700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f794014d8c0 0x7f7940064440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37212/0 (socket says 172.21.15.88:37212) 2023-10-18T16:09:41.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.767+0000 7f793f7fe700 1 -- 172.21.15.88:0/481179064 learned_addr learned my addr 172.21.15.88:0/481179064 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:09:41.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.767+0000 7f793effd700 1 --2- 172.21.15.88:0/481179064 >> v2:172.21.15.151:3300/0 conn(0x7f79400a9050 0x7f7940064930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:41.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.768+0000 7f793f7fe700 1 -- 172.21.15.88:0/481179064 >> v2:172.21.15.151:3300/0 conn(0x7f79400a9050 msgr2=0x7f7940064930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:09:41.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.768+0000 7f793f7fe700 1 --2- 172.21.15.88:0/481179064 >> v2:172.21.15.151:3300/0 conn(0x7f79400a9050 0x7f7940064930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:41.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.768+0000 7f793f7fe700 1 -- 172.21.15.88:0/481179064 >> v1:172.21.15.151:6789/0 conn(0x7f7940065360 legacy=0x7f7940153ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:41.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.768+0000 7f793f7fe700 1 -- 172.21.15.88:0/481179064 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79400c72c0 con 0x7f794014d8c0 2023-10-18T16:09:41.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.768+0000 7f793f7fe700 1 --2- 172.21.15.88:0/481179064 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f794014d8c0 0x7f7940064440 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f79400637d0 tx=0x7f7930009840).ready entity=mon.0 client_cookie=8a913db460918d5b server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:09:41.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.768+0000 7f793e7fc700 1 -- 172.21.15.88:0/481179064 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f793000e070 con 0x7f794014d8c0 2023-10-18T16:09:41.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.768+0000 7f793e7fc700 1 -- 172.21.15.88:0/481179064 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f793000ba50 con 0x7f794014d8c0 2023-10-18T16:09:41.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.768+0000 7f793e7fc700 1 -- 172.21.15.88:0/481179064 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f793000bdd0 con 0x7f794014d8c0 2023-10-18T16:09:41.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.768+0000 7f7945e06700 1 -- 172.21.15.88:0/481179064 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f794014d8c0 msgr2=0x7f7940064440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:41.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.768+0000 7f7945e06700 1 --2- 172.21.15.88:0/481179064 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f794014d8c0 0x7f7940064440 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f79400637d0 tx=0x7f7930009840).stop 2023-10-18T16:09:41.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.768+0000 7f7945e06700 1 -- 172.21.15.88:0/481179064 shutdown_connections 2023-10-18T16:09:41.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.768+0000 7f7945e06700 1 --2- 172.21.15.88:0/481179064 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f794014d8c0 0x7f7940064440 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:41.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.768+0000 7f7945e06700 1 --2- 172.21.15.88:0/481179064 >> v2:172.21.15.151:3300/0 conn(0x7f79400a9050 0x7f7940064930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:41.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.768+0000 7f7945e06700 1 -- 172.21.15.88:0/481179064 >> 172.21.15.88:0/481179064 conn(0x7f794006a810 msgr2=0x7f79400c6a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:09:41.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.768+0000 7f7945e06700 1 -- 172.21.15.88:0/481179064 shutdown_connections 2023-10-18T16:09:41.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.768+0000 7f7945e06700 1 -- 172.21.15.88:0/481179064 wait complete. 2023-10-18T16:09:41.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.770+0000 7f7945e06700 1 Processor -- start 2023-10-18T16:09:41.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.770+0000 7f7945e06700 1 -- start start 2023-10-18T16:09:41.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.770+0000 7f7945e06700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7940065360 0x7f7940150e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:41.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.770+0000 7f7945e06700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f79400a9050 0x7f79400a25c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:41.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.770+0000 7f7945e06700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f794014d8c0 0x7f794014fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:41.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.770+0000 7f7945e06700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7940164930 con 0x7f79400a9050 2023-10-18T16:09:41.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.770+0000 7f7945e06700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f794014f780 con 0x7f794014d8c0 2023-10-18T16:09:41.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.770+0000 7f7945e06700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7940163db0 con 0x7f7940065360 2023-10-18T16:09:41.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.770+0000 7f793effd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f79400a9050 0x7f79400a25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:41.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.771+0000 7f793effd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f79400a9050 0x7f79400a25c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37216/0 (socket says 172.21.15.88:37216) 2023-10-18T16:09:41.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.771+0000 7f793effd700 1 -- 172.21.15.88:0/2000636994 learned_addr learned my addr 172.21.15.88:0/2000636994 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:09:41.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.771+0000 7f793ffff700 1 --2- 172.21.15.88:0/2000636994 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f794014d8c0 0x7f794014fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:41.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.771+0000 7f793f7fe700 1 --2- 172.21.15.88:0/2000636994 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7940065360 0x7f7940150e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:41.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.771+0000 7f793effd700 1 -- 172.21.15.88:0/2000636994 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7940065360 msgr2=0x7f7940150e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:41.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.771+0000 7f793effd700 1 --2- 172.21.15.88:0/2000636994 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7940065360 0x7f7940150e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:41.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.771+0000 7f793effd700 1 -- 172.21.15.88:0/2000636994 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f794014d8c0 msgr2=0x7f794014fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:41.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.771+0000 7f793effd700 1 --2- 172.21.15.88:0/2000636994 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f794014d8c0 0x7f794014fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:41.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.771+0000 7f793effd700 1 -- 172.21.15.88:0/2000636994 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7930009ae0 con 0x7f79400a9050 2023-10-18T16:09:41.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.771+0000 7f793f7fe700 1 --2- 172.21.15.88:0/2000636994 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7940065360 0x7f7940150e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-10-18T16:09:41.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.771+0000 7f793effd700 1 --2- 172.21.15.88:0/2000636994 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f79400a9050 0x7f79400a25c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f792800d8c0 tx=0x7f7928008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:09:41.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.771+0000 7f79377fe700 1 -- 172.21.15.88:0/2000636994 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7928009640 con 0x7f79400a9050 2023-10-18T16:09:41.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.771+0000 7f79377fe700 1 -- 172.21.15.88:0/2000636994 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79280097a0 con 0x7f79400a9050 2023-10-18T16:09:41.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.771+0000 7f7945e06700 1 -- 172.21.15.88:0/2000636994 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7940164090 con 0x7f79400a9050 2023-10-18T16:09:41.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.771+0000 7f79377fe700 1 -- 172.21.15.88:0/2000636994 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7928009b20 con 0x7f79400a9050 2023-10-18T16:09:41.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.771+0000 7f7945e06700 1 -- 172.21.15.88:0/2000636994 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7940164650 con 0x7f79400a9050 2023-10-18T16:09:41.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.772+0000 7f7945e06700 1 -- 172.21.15.88:0/2000636994 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f794014f910 con 0x7f79400a9050 2023-10-18T16:09:41.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.773+0000 7f79377fe700 1 -- 172.21.15.88:0/2000636994 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f7928013480 con 0x7f79400a9050 2023-10-18T16:09:41.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.773+0000 7f79377fe700 1 --2- 172.21.15.88:0/2000636994 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f792005f120 0x7f79200615e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:41.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.773+0000 7f79377fe700 1 -- 172.21.15.88:0/2000636994 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6725+0+0 (secure 0 0 0) 0x7f792807a2b0 con 0x7f79400a9050 2023-10-18T16:09:41.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.773+0000 7f793f7fe700 1 --2- 172.21.15.88:0/2000636994 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f792005f120 0x7f79200615e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:41.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.773+0000 7f793f7fe700 1 --2- 172.21.15.88:0/2000636994 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f792005f120 0x7f79200615e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f7930009930 tx=0x7f7930009840).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:09:41.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.777+0000 7f79377fe700 1 -- 172.21.15.88:0/2000636994 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f792804b2c0 con 0x7f79400a9050 2023-10-18T16:09:41.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.978+0000 7f7945e06700 1 -- 172.21.15.88:0/2000636994 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f79400c75a0 con 0x7f79400a9050 2023-10-18T16:09:41.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.979+0000 7f79377fe700 1 -- 172.21.15.88:0/2000636994 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7f792804e8e0 con 0x7f79400a9050 2023-10-18T16:09:41.983 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:09:41.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.980+0000 7f7945e06700 1 -- 172.21.15.88:0/2000636994 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f792005f120 msgr2=0x7f79200615e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:41.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.980+0000 7f7945e06700 1 --2- 172.21.15.88:0/2000636994 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f792005f120 0x7f79200615e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f7930009930 tx=0x7f7930009840).stop 2023-10-18T16:09:41.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.980+0000 7f7945e06700 1 -- 172.21.15.88:0/2000636994 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f79400a9050 msgr2=0x7f79400a25c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:41.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.980+0000 7f7945e06700 1 --2- 172.21.15.88:0/2000636994 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f79400a9050 0x7f79400a25c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f792800d8c0 tx=0x7f7928008040).stop 2023-10-18T16:09:41.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.980+0000 7f7945e06700 1 -- 172.21.15.88:0/2000636994 shutdown_connections 2023-10-18T16:09:41.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.980+0000 7f7945e06700 1 --2- 172.21.15.88:0/2000636994 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f792005f120 0x7f79200615e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:41.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.980+0000 7f7945e06700 1 --2- 172.21.15.88:0/2000636994 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f794014d8c0 0x7f794014fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:41.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.980+0000 7f7945e06700 1 --2- 172.21.15.88:0/2000636994 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7940065360 0x7f7940150e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:41.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.980+0000 7f7945e06700 1 --2- 172.21.15.88:0/2000636994 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f79400a9050 0x7f79400a25c0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:41.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.980+0000 7f7945e06700 1 -- 172.21.15.88:0/2000636994 >> 172.21.15.88:0/2000636994 conn(0x7f794006a810 msgr2=0x7f79400c6420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:09:41.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.980+0000 7f7945e06700 1 -- 172.21.15.88:0/2000636994 shutdown_connections 2023-10-18T16:09:41.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:41.980+0000 7f7945e06700 1 -- 172.21.15.88:0/2000636994 wait complete. 2023-10-18T16:09:41.996 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:09:42.197 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~6s 2023-10-18T16:09:42.197 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~19s 2023-10-18T16:09:42.198 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~52s 2023-10-18T16:09:42.198 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~32s 2023-10-18T16:09:42.198 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~113s 2023-10-18T16:09:42.198 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.f is failed for ~65s 2023-10-18T16:09:44.997 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:09:45.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.117+0000 7f40e2902700 1 Processor -- start 2023-10-18T16:09:45.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.117+0000 7f40e2902700 1 -- start start 2023-10-18T16:09:45.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.117+0000 7f40e2902700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f40dc0a9050 0x7f40dc153ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:45.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.117+0000 7f40e2902700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f40dc065360 0x7f40dc1543c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:45.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.117+0000 7f40e2902700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40dc064150 con 0x7f40dc14d8c0 2023-10-18T16:09:45.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.117+0000 7f40e2902700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40dc153a40 con 0x7f40dc0a9050 2023-10-18T16:09:45.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.117+0000 7f40e2902700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40dc063e70 con 0x7f40dc065360 2023-10-18T16:09:45.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.118+0000 7f40db7fe700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f40dc0a9050 0x7f40dc153ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:45.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.118+0000 7f40db7fe700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f40dc0a9050 0x7f40dc153ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37222/0 (socket says 172.21.15.88:37222) 2023-10-18T16:09:45.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.118+0000 7f40db7fe700 1 -- 172.21.15.88:0/2785846391 learned_addr learned my addr 172.21.15.88:0/2785846391 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:09:45.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.118+0000 7f40e0e9f700 1 --2- 172.21.15.88:0/2785846391 >> v2:172.21.15.151:3300/0 conn(0x7f40dc065360 0x7f40dc1543c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:45.125 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.118+0000 7f40db7fe700 1 -- 172.21.15.88:0/2785846391 >> v2:172.21.15.151:3300/0 conn(0x7f40dc065360 msgr2=0x7f40dc1543c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:09:45.125 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.118+0000 7f40db7fe700 1 --2- 172.21.15.88:0/2785846391 >> v2:172.21.15.151:3300/0 conn(0x7f40dc065360 0x7f40dc1543c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:45.125 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.118+0000 7f40db7fe700 1 -- 172.21.15.88:0/2785846391 >> v1:172.21.15.151:6789/0 conn(0x7f40dc14d8c0 legacy=0x7f40dc064440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:45.125 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.118+0000 7f40db7fe700 1 -- 172.21.15.88:0/2785846391 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40dc0c72c0 con 0x7f40dc0a9050 2023-10-18T16:09:45.125 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.118+0000 7f40db7fe700 1 --2- 172.21.15.88:0/2785846391 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40dc0a9050 0x7f40dc153ed0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f40dc0c4950 tx=0x7f40cc0090f0).ready entity=mon.0 client_cookie=7b8b60c408d0d0a server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:09:45.126 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.118+0000 7f40daffd700 1 -- 172.21.15.88:0/2785846391 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40cc00b600 con 0x7f40dc0a9050 2023-10-18T16:09:45.126 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.118+0000 7f40daffd700 1 -- 172.21.15.88:0/2785846391 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40cc00b760 con 0x7f40dc0a9050 2023-10-18T16:09:45.126 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.118+0000 7f40daffd700 1 -- 172.21.15.88:0/2785846391 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40cc00bb00 con 0x7f40dc0a9050 2023-10-18T16:09:45.126 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.119+0000 7f40e2902700 1 -- 172.21.15.88:0/2785846391 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40dc0a9050 msgr2=0x7f40dc153ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:45.127 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.119+0000 7f40e2902700 1 --2- 172.21.15.88:0/2785846391 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40dc0a9050 0x7f40dc153ed0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f40dc0c4950 tx=0x7f40cc0090f0).stop 2023-10-18T16:09:45.127 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.119+0000 7f40e2902700 1 -- 172.21.15.88:0/2785846391 shutdown_connections 2023-10-18T16:09:45.127 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.119+0000 7f40e2902700 1 --2- 172.21.15.88:0/2785846391 >> v2:172.21.15.151:3300/0 conn(0x7f40dc065360 0x7f40dc1543c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:45.127 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.119+0000 7f40e2902700 1 --2- 172.21.15.88:0/2785846391 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40dc0a9050 0x7f40dc153ed0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:45.127 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.119+0000 7f40e2902700 1 -- 172.21.15.88:0/2785846391 >> 172.21.15.88:0/2785846391 conn(0x7f40dc06a810 msgr2=0x7f40dc0c6a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:09:45.127 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.119+0000 7f40e2902700 1 -- 172.21.15.88:0/2785846391 shutdown_connections 2023-10-18T16:09:45.128 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.119+0000 7f40e2902700 1 -- 172.21.15.88:0/2785846391 wait complete. 2023-10-18T16:09:45.128 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.120+0000 7f40e2902700 1 Processor -- start 2023-10-18T16:09:45.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.120+0000 7f40e2902700 1 -- start start 2023-10-18T16:09:45.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.121+0000 7f40e2902700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40dc065360 0x7f40dc160b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:45.131 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.121+0000 7f40e2902700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40dc0a9050 0x7f40dc14c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:45.131 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.121+0000 7f40e2902700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40dc14d8c0 0x7f40dc1509b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:45.131 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.121+0000 7f40e2902700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40dc14e6e0 con 0x7f40dc065360 2023-10-18T16:09:45.131 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.121+0000 7f40e2902700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40dc160700 con 0x7f40dc14d8c0 2023-10-18T16:09:45.131 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.121+0000 7f40e2902700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40dc14f050 con 0x7f40dc0a9050 2023-10-18T16:09:45.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.121+0000 7f40dbfff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40dc065360 0x7f40dc160b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:45.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.121+0000 7f40dbfff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40dc065360 0x7f40dc160b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37224/0 (socket says 172.21.15.88:37224) 2023-10-18T16:09:45.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.121+0000 7f40dbfff700 1 -- 172.21.15.88:0/1220175001 learned_addr learned my addr 172.21.15.88:0/1220175001 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:09:45.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.121+0000 7f40e0e9f700 1 --2- 172.21.15.88:0/1220175001 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40dc14d8c0 0x7f40dc1509b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:45.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.121+0000 7f40db7fe700 1 --2- 172.21.15.88:0/1220175001 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40dc0a9050 0x7f40dc14c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:45.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.121+0000 7f40dbfff700 1 -- 172.21.15.88:0/1220175001 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40dc0a9050 msgr2=0x7f40dc14c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:45.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.121+0000 7f40dbfff700 1 --2- 172.21.15.88:0/1220175001 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40dc0a9050 0x7f40dc14c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:45.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.121+0000 7f40dbfff700 1 -- 172.21.15.88:0/1220175001 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40dc14d8c0 msgr2=0x7f40dc1509b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:45.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.121+0000 7f40dbfff700 1 --2- 172.21.15.88:0/1220175001 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40dc14d8c0 0x7f40dc1509b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:45.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.121+0000 7f40dbfff700 1 -- 172.21.15.88:0/1220175001 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40cc009370 con 0x7f40dc065360 2023-10-18T16:09:45.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.121+0000 7f40dbfff700 1 --2- 172.21.15.88:0/1220175001 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40dc065360 0x7f40dc160b90 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f40d000c470 tx=0x7f40d0007db0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:09:45.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.121+0000 7f40c3fff700 1 -- 172.21.15.88:0/1220175001 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40d000ce90 con 0x7f40dc065360 2023-10-18T16:09:45.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.121+0000 7f40c3fff700 1 -- 172.21.15.88:0/1220175001 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40d000f470 con 0x7f40dc065360 2023-10-18T16:09:45.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.121+0000 7f40e2902700 1 -- 172.21.15.88:0/1220175001 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40dc0c7960 con 0x7f40dc065360 2023-10-18T16:09:45.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.121+0000 7f40c3fff700 1 -- 172.21.15.88:0/1220175001 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40d000f810 con 0x7f40dc065360 2023-10-18T16:09:45.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.121+0000 7f40e2902700 1 -- 172.21.15.88:0/1220175001 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40dc14ed70 con 0x7f40dc065360 2023-10-18T16:09:45.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.122+0000 7f40e2902700 1 -- 172.21.15.88:0/1220175001 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40dc06c4e0 con 0x7f40dc065360 2023-10-18T16:09:45.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.123+0000 7f40c3fff700 1 -- 172.21.15.88:0/1220175001 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f40d000fbc0 con 0x7f40dc065360 2023-10-18T16:09:45.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.123+0000 7f40c3fff700 1 --2- 172.21.15.88:0/1220175001 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f40c805f120 0x7f40c80615e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:45.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.123+0000 7f40c3fff700 1 -- 172.21.15.88:0/1220175001 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6725+0+0 (secure 0 0 0) 0x7f40d0079930 con 0x7f40dc065360 2023-10-18T16:09:45.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.123+0000 7f40db7fe700 1 --2- 172.21.15.88:0/1220175001 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f40c805f120 0x7f40c80615e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:45.136 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.124+0000 7f40db7fe700 1 --2- 172.21.15.88:0/1220175001 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f40c805f120 0x7f40c80615e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f40dc1637b0 tx=0x7f40cc017000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:09:45.136 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.127+0000 7f40c3fff700 1 -- 172.21.15.88:0/1220175001 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f40d004a9a0 con 0x7f40dc065360 2023-10-18T16:09:45.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.330+0000 7f40e2902700 1 -- 172.21.15.88:0/1220175001 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f40dc0c75a0 con 0x7f40dc065360 2023-10-18T16:09:45.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.330+0000 7f40c3fff700 1 -- 172.21.15.88:0/1220175001 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7f40d004dfc0 con 0x7f40dc065360 2023-10-18T16:09:45.334 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:09:45.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.332+0000 7f40e2902700 1 -- 172.21.15.88:0/1220175001 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f40c805f120 msgr2=0x7f40c80615e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:45.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.332+0000 7f40e2902700 1 --2- 172.21.15.88:0/1220175001 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f40c805f120 0x7f40c80615e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f40dc1637b0 tx=0x7f40cc017000).stop 2023-10-18T16:09:45.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.332+0000 7f40e2902700 1 -- 172.21.15.88:0/1220175001 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40dc065360 msgr2=0x7f40dc160b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:45.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.332+0000 7f40e2902700 1 --2- 172.21.15.88:0/1220175001 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40dc065360 0x7f40dc160b90 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f40d000c470 tx=0x7f40d0007db0).stop 2023-10-18T16:09:45.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.332+0000 7f40e2902700 1 -- 172.21.15.88:0/1220175001 shutdown_connections 2023-10-18T16:09:45.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.332+0000 7f40e2902700 1 --2- 172.21.15.88:0/1220175001 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f40c805f120 0x7f40c80615e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:45.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.332+0000 7f40e2902700 1 --2- 172.21.15.88:0/1220175001 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40dc14d8c0 0x7f40dc1509b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:45.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.332+0000 7f40e2902700 1 --2- 172.21.15.88:0/1220175001 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40dc065360 0x7f40dc160b90 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:45.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.332+0000 7f40e2902700 1 --2- 172.21.15.88:0/1220175001 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40dc0a9050 0x7f40dc14c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:45.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.332+0000 7f40e2902700 1 -- 172.21.15.88:0/1220175001 >> 172.21.15.88:0/1220175001 conn(0x7f40dc06a810 msgr2=0x7f40dc0c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:09:45.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.332+0000 7f40e2902700 1 -- 172.21.15.88:0/1220175001 shutdown_connections 2023-10-18T16:09:45.338 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:45.332+0000 7f40e2902700 1 -- 172.21.15.88:0/1220175001 wait complete. 2023-10-18T16:09:45.347 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:09:48.348 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:09:48.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.471+0000 7f1b02366700 1 Processor -- start 2023-10-18T16:09:48.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.471+0000 7f1b02366700 1 -- start start 2023-10-18T16:09:48.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.472+0000 7f1b02366700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f1afc153010 0x7f1afc1533f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:48.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.472+0000 7f1b02366700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1afc1525f0 0x7f1afc0c36c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:48.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.472+0000 7f1b02366700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1afc0a9670 con 0x7f1afc064d70 2023-10-18T16:09:48.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.472+0000 7f1b02366700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1afc14df50 con 0x7f1afc1525f0 2023-10-18T16:09:48.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.472+0000 7f1b02366700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1afc151a50 con 0x7f1afc153010 2023-10-18T16:09:48.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.472+0000 7f1b00903700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1afc1525f0 0x7f1afc0c36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:48.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.472+0000 7f1afb7fe700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f1afc064d70 0x7f1afc151ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:47988/0 (socket says 172.21.15.88:47988) 2023-10-18T16:09:48.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.472+0000 7f1b00903700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1afc1525f0 0x7f1afc0c36c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:53394/0 (socket says 172.21.15.88:53394) 2023-10-18T16:09:48.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.472+0000 7f1afb7fe700 1 -- 172.21.15.88:0/2366571168 learned_addr learned my addr 172.21.15.88:0/2366571168 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:09:48.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.472+0000 7f1afbfff700 1 --2- 172.21.15.88:0/2366571168 >> v2:172.21.15.151:3300/0 conn(0x7f1afc153010 0x7f1afc1533f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:48.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.472+0000 7f1b00903700 1 -- 172.21.15.88:0/2366571168 >> v2:172.21.15.151:3300/0 conn(0x7f1afc153010 msgr2=0x7f1afc1533f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:09:48.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.472+0000 7f1b00903700 1 --2- 172.21.15.88:0/2366571168 >> v2:172.21.15.151:3300/0 conn(0x7f1afc153010 0x7f1afc1533f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:48.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.472+0000 7f1b00903700 1 -- 172.21.15.88:0/2366571168 >> v1:172.21.15.88:6789/0 conn(0x7f1afc064d70 legacy=0x7f1afc151ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:48.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.472+0000 7f1b00903700 1 -- 172.21.15.88:0/2366571168 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1afc0a8af0 con 0x7f1afc1525f0 2023-10-18T16:09:48.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.472+0000 7f1afaffd700 1 -- 172.21.15.88:0/2366571168 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 74537133 0 0) 0x7f1afc0a9670 con 0x7f1afc064d70 2023-10-18T16:09:48.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.472+0000 7f1b00903700 1 --2- 172.21.15.88:0/2366571168 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1afc1525f0 0x7f1afc0c36c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f1afc06aaa0 tx=0x7f1af4009780).ready entity=mon.0 client_cookie=13990ea6fcbe0707 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:09:48.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.472+0000 7f1afaffd700 1 -- 172.21.15.88:0/2366571168 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1af400e070 con 0x7f1afc1525f0 2023-10-18T16:09:48.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.472+0000 7f1afaffd700 1 -- 172.21.15.88:0/2366571168 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1af400bc30 con 0x7f1afc1525f0 2023-10-18T16:09:48.481 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.472+0000 7f1afaffd700 1 -- 172.21.15.88:0/2366571168 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1af4012410 con 0x7f1afc1525f0 2023-10-18T16:09:48.481 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.473+0000 7f1b02366700 1 -- 172.21.15.88:0/2366571168 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1afc1525f0 msgr2=0x7f1afc0c36c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:48.481 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.473+0000 7f1b02366700 1 --2- 172.21.15.88:0/2366571168 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1afc1525f0 0x7f1afc0c36c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f1afc06aaa0 tx=0x7f1af4009780).stop 2023-10-18T16:09:48.481 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.473+0000 7f1b02366700 1 -- 172.21.15.88:0/2366571168 shutdown_connections 2023-10-18T16:09:48.481 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.473+0000 7f1b02366700 1 --2- 172.21.15.88:0/2366571168 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1afc1525f0 0x7f1afc0c36c0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:48.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.473+0000 7f1b02366700 1 --2- 172.21.15.88:0/2366571168 >> v2:172.21.15.151:3300/0 conn(0x7f1afc153010 0x7f1afc1533f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:48.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.473+0000 7f1b02366700 1 -- 172.21.15.88:0/2366571168 >> 172.21.15.88:0/2366571168 conn(0x7f1afc069d80 msgr2=0x7f1afc06a190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:09:48.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.473+0000 7f1b02366700 1 -- 172.21.15.88:0/2366571168 shutdown_connections 2023-10-18T16:09:48.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.473+0000 7f1b02366700 1 -- 172.21.15.88:0/2366571168 wait complete. 2023-10-18T16:09:48.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.474+0000 7f1b02366700 1 Processor -- start 2023-10-18T16:09:48.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.474+0000 7f1b02366700 1 -- start start 2023-10-18T16:09:48.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.475+0000 7f1b02366700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1afc064d70 0x7f1afc14ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:48.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.475+0000 7f1b02366700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1afc1525f0 0x7f1afc0c02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:48.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.475+0000 7f1b02366700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1afc153010 0x7f1afc0ab3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:48.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.475+0000 7f1b02366700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1afc0bf540 con 0x7f1afc064d70 2023-10-18T16:09:48.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.475+0000 7f1b02366700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1afc151700 con 0x7f1afc153010 2023-10-18T16:09:48.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.475+0000 7f1b02366700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1afc14f880 con 0x7f1afc1525f0 2023-10-18T16:09:48.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.475+0000 7f1afbfff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1afc064d70 0x7f1afc14ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:48.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.475+0000 7f1afbfff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1afc064d70 0x7f1afc14ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:53398/0 (socket says 172.21.15.88:53398) 2023-10-18T16:09:48.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.475+0000 7f1afbfff700 1 -- 172.21.15.88:0/1250818109 learned_addr learned my addr 172.21.15.88:0/1250818109 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:09:48.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.475+0000 7f1b00903700 1 --2- 172.21.15.88:0/1250818109 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1afc153010 0x7f1afc0ab3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:48.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.475+0000 7f1afbfff700 1 -- 172.21.15.88:0/1250818109 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1afc1525f0 msgr2=0x7f1afc0c02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:48.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.475+0000 7f1afbfff700 1 --2- 172.21.15.88:0/1250818109 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1afc1525f0 0x7f1afc0c02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-10-18T16:09:48.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.475+0000 7f1afbfff700 1 -- 172.21.15.88:0/1250818109 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1afc153010 msgr2=0x7f1afc0ab3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:48.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.475+0000 7f1afbfff700 1 --2- 172.21.15.88:0/1250818109 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1afc153010 0x7f1afc0ab3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:48.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.476+0000 7f1afbfff700 1 -- 172.21.15.88:0/1250818109 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1af4009a20 con 0x7f1afc064d70 2023-10-18T16:09:48.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.476+0000 7f1b00903700 1 --2- 172.21.15.88:0/1250818109 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1afc153010 0x7f1afc0ab3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-10-18T16:09:48.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.476+0000 7f1afbfff700 1 --2- 172.21.15.88:0/1250818109 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1afc064d70 0x7f1afc14ecf0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f1aec00d890 tx=0x7f1aec008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:09:48.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.476+0000 7f1ae3fff700 1 -- 172.21.15.88:0/1250818109 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aec0125f0 con 0x7f1afc064d70 2023-10-18T16:09:48.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.476+0000 7f1ae3fff700 1 -- 172.21.15.88:0/1250818109 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1aec012750 con 0x7f1afc064d70 2023-10-18T16:09:48.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.476+0000 7f1ae3fff700 1 -- 172.21.15.88:0/1250818109 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aec012ad0 con 0x7f1afc064d70 2023-10-18T16:09:48.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.476+0000 7f1b02366700 1 -- 172.21.15.88:0/1250818109 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1afc0aaf40 con 0x7f1afc064d70 2023-10-18T16:09:48.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.476+0000 7f1b02366700 1 -- 172.21.15.88:0/1250818109 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1afc0be3a0 con 0x7f1afc064d70 2023-10-18T16:09:48.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.477+0000 7f1b02366700 1 -- 172.21.15.88:0/1250818109 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1afc152320 con 0x7f1afc064d70 2023-10-18T16:09:48.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.477+0000 7f1ae3fff700 1 -- 172.21.15.88:0/1250818109 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f1aec013480 con 0x7f1afc064d70 2023-10-18T16:09:48.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.478+0000 7f1ae3fff700 1 --2- 172.21.15.88:0/1250818109 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f1ae805f150 0x7f1ae8061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:48.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.478+0000 7f1ae3fff700 1 -- 172.21.15.88:0/1250818109 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6725+0+0 (secure 0 0 0) 0x7f1aec07a2b0 con 0x7f1afc064d70 2023-10-18T16:09:48.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.478+0000 7f1afb7fe700 1 --2- 172.21.15.88:0/1250818109 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f1ae805f150 0x7f1ae8061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:48.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.478+0000 7f1afb7fe700 1 --2- 172.21.15.88:0/1250818109 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f1ae805f150 0x7f1ae8061610 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f1af00011e0 tx=0x7f1af0007850).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:09:48.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.482+0000 7f1ae3fff700 1 -- 172.21.15.88:0/1250818109 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f1afc152320 con 0x7f1afc064d70 2023-10-18T16:09:48.508 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~13s 2023-10-18T16:09:48.509 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~25s 2023-10-18T16:09:48.509 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~58s 2023-10-18T16:09:48.509 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~38s 2023-10-18T16:09:48.509 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~119s 2023-10-18T16:09:48.509 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.f is failed for ~72s 2023-10-18T16:09:48.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.681+0000 7f1b02366700 1 -- 172.21.15.88:0/1250818109 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f1afc0be640 con 0x7f1afc064d70 2023-10-18T16:09:48.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.681+0000 7f1ae3fff700 1 -- 172.21.15.88:0/1250818109 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7f1aec04b2c0 con 0x7f1afc064d70 2023-10-18T16:09:48.685 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:09:48.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.683+0000 7f1b02366700 1 -- 172.21.15.88:0/1250818109 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f1ae805f150 msgr2=0x7f1ae8061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:48.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.683+0000 7f1b02366700 1 --2- 172.21.15.88:0/1250818109 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f1ae805f150 0x7f1ae8061610 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f1af00011e0 tx=0x7f1af0007850).stop 2023-10-18T16:09:48.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.683+0000 7f1b02366700 1 -- 172.21.15.88:0/1250818109 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1afc064d70 msgr2=0x7f1afc14ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:48.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.683+0000 7f1b02366700 1 --2- 172.21.15.88:0/1250818109 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1afc064d70 0x7f1afc14ecf0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f1aec00d890 tx=0x7f1aec008040).stop 2023-10-18T16:09:48.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.683+0000 7f1b02366700 1 -- 172.21.15.88:0/1250818109 shutdown_connections 2023-10-18T16:09:48.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.683+0000 7f1b02366700 1 --2- 172.21.15.88:0/1250818109 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f1ae805f150 0x7f1ae8061610 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:48.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.683+0000 7f1b02366700 1 --2- 172.21.15.88:0/1250818109 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1afc153010 0x7f1afc0ab3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:48.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.683+0000 7f1b02366700 1 --2- 172.21.15.88:0/1250818109 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1afc064d70 0x7f1afc14ecf0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:48.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.683+0000 7f1b02366700 1 --2- 172.21.15.88:0/1250818109 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1afc1525f0 0x7f1afc0c02b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-10-18T16:09:48.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.683+0000 7f1b02366700 1 -- 172.21.15.88:0/1250818109 >> 172.21.15.88:0/1250818109 conn(0x7f1afc069d80 msgr2=0x7f1afc0ba900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:09:48.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.683+0000 7f1b02366700 1 -- 172.21.15.88:0/1250818109 shutdown_connections 2023-10-18T16:09:48.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:48.683+0000 7f1b02366700 1 -- 172.21.15.88:0/1250818109 wait complete. 2023-10-18T16:09:48.698 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:09:51.699 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:09:51.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:51.820+0000 7f3d81d67700 1 Processor -- start 2023-10-18T16:09:51.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:51.820+0000 7f3d81d67700 1 -- start start 2023-10-18T16:09:51.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:51.821+0000 7f3d81d67700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3d7c14d8c0 0x7f3d7c064440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:51.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:51.821+0000 7f3d81d67700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3d7c0a9050 0x7f3d7c064930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:51.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:51.821+0000 7f3d81d67700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d7c064150 con 0x7f3d7c065360 2023-10-18T16:09:51.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:51.821+0000 7f3d81d67700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d7c153a40 con 0x7f3d7c0a9050 2023-10-18T16:09:51.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:51.821+0000 7f3d81d67700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d7c063e70 con 0x7f3d7c14d8c0 2023-10-18T16:09:51.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:51.821+0000 7f3d7affd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3d7c0a9050 0x7f3d7c064930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:51.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:51.821+0000 7f3d7affd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3d7c0a9050 0x7f3d7c064930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:53406/0 (socket says 172.21.15.88:53406) 2023-10-18T16:09:51.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:51.821+0000 7f3d7affd700 1 -- 172.21.15.88:0/2293663427 learned_addr learned my addr 172.21.15.88:0/2293663427 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:09:51.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:51.821+0000 7f3d7b7fe700 1 --2- 172.21.15.88:0/2293663427 >> v2:172.21.15.151:3300/0 conn(0x7f3d7c14d8c0 0x7f3d7c064440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:51.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:51.821+0000 7f3d7affd700 1 -- 172.21.15.88:0/2293663427 >> v2:172.21.15.151:3300/0 conn(0x7f3d7c14d8c0 msgr2=0x7f3d7c064440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:09:51.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:51.821+0000 7f3d7affd700 1 --2- 172.21.15.88:0/2293663427 >> v2:172.21.15.151:3300/0 conn(0x7f3d7c14d8c0 0x7f3d7c064440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:51.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:51.821+0000 7f3d7affd700 1 -- 172.21.15.88:0/2293663427 >> v1:172.21.15.151:6789/0 conn(0x7f3d7c065360 legacy=0x7f3d7c153ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:51.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:51.821+0000 7f3d7affd700 1 -- 172.21.15.88:0/2293663427 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d7c0c72c0 con 0x7f3d7c0a9050 2023-10-18T16:09:51.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:51.821+0000 7f3d7b7fe700 1 --2- 172.21.15.88:0/2293663427 >> v2:172.21.15.151:3300/0 conn(0x7f3d7c14d8c0 0x7f3d7c064440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:09:51.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:51.821+0000 7f3d7affd700 1 --2- 172.21.15.88:0/2293663427 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3d7c0a9050 0x7f3d7c064930 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f3d7c062d50 tx=0x7f3d700090f0).ready entity=mon.0 client_cookie=f2de01a44f07724f server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:09:51.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:51.822+0000 7f3d7a7fc700 1 -- 172.21.15.88:0/2293663427 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d7000b600 con 0x7f3d7c0a9050 2023-10-18T16:09:51.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:51.822+0000 7f3d7a7fc700 1 -- 172.21.15.88:0/2293663427 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d7000b760 con 0x7f3d7c0a9050 2023-10-18T16:09:51.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:51.822+0000 7f3d7a7fc700 1 -- 172.21.15.88:0/2293663427 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d7000bb00 con 0x7f3d7c0a9050 2023-10-18T16:09:51.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:51.822+0000 7f3d81d67700 1 -- 172.21.15.88:0/2293663427 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3d7c0a9050 msgr2=0x7f3d7c064930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:51.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:51.822+0000 7f3d81d67700 1 --2- 172.21.15.88:0/2293663427 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3d7c0a9050 0x7f3d7c064930 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f3d7c062d50 tx=0x7f3d700090f0).stop 2023-10-18T16:09:51.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:51.822+0000 7f3d81d67700 1 -- 172.21.15.88:0/2293663427 shutdown_connections 2023-10-18T16:09:51.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:51.822+0000 7f3d81d67700 1 --2- 172.21.15.88:0/2293663427 >> v2:172.21.15.151:3300/0 conn(0x7f3d7c14d8c0 0x7f3d7c064440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:51.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:51.822+0000 7f3d81d67700 1 --2- 172.21.15.88:0/2293663427 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3d7c0a9050 0x7f3d7c064930 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:51.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:51.822+0000 7f3d81d67700 1 -- 172.21.15.88:0/2293663427 >> 172.21.15.88:0/2293663427 conn(0x7f3d7c06a810 msgr2=0x7f3d7c0c6a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:09:51.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:51.822+0000 7f3d81d67700 1 -- 172.21.15.88:0/2293663427 shutdown_connections 2023-10-18T16:09:51.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:51.822+0000 7f3d81d67700 1 -- 172.21.15.88:0/2293663427 wait complete. 2023-10-18T16:09:51.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:51.823+0000 7f3d81d67700 1 Processor -- start 2023-10-18T16:09:51.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:51.823+0000 7f3d81d67700 1 -- start start 2023-10-18T16:09:51.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:51.824+0000 7f3d81d67700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3d7c065360 0x7f3d7c0beac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:51.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:51.824+0000 7f3d81d67700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3d7c0a9050 0x7f3d7c0ac450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:51.835 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:51.824+0000 7f3d81d67700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3d7c14d8c0 0x7f3d7c0c0360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:51.835 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:51.824+0000 7f3d81d67700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d7c160580 con 0x7f3d7c14d8c0 2023-10-18T16:09:51.835 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:51.824+0000 7f3d81d67700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d7c149a20 con 0x7f3d7c065360 2023-10-18T16:09:51.835 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:51.824+0000 7f3d81d67700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d7c0bd5b0 con 0x7f3d7c0a9050 2023-10-18T16:09:51.836 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:51.824+0000 7f3d7bfff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3d7c14d8c0 0x7f3d7c0c0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:51.836 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:51.824+0000 7f3d7bfff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3d7c14d8c0 0x7f3d7c0c0360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:53418/0 (socket says 172.21.15.88:53418) 2023-10-18T16:09:51.836 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:51.824+0000 7f3d7bfff700 1 -- 172.21.15.88:0/2548686352 learned_addr learned my addr 172.21.15.88:0/2548686352 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:09:51.836 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:51.824+0000 7f3d7b7fe700 1 --2- 172.21.15.88:0/2548686352 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3d7c065360 0x7f3d7c0beac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:51.836 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:51.824+0000 7f3d7affd700 1 --2- 172.21.15.88:0/2548686352 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3d7c0a9050 0x7f3d7c0ac450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:51.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:51.825+0000 7f3d7bfff700 1 -- 172.21.15.88:0/2548686352 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3d7c0a9050 msgr2=0x7f3d7c0ac450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:51.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:51.825+0000 7f3d7bfff700 1 --2- 172.21.15.88:0/2548686352 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3d7c0a9050 0x7f3d7c0ac450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:51.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:51.825+0000 7f3d7bfff700 1 -- 172.21.15.88:0/2548686352 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3d7c065360 msgr2=0x7f3d7c0beac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:51.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:51.825+0000 7f3d7bfff700 1 --2- 172.21.15.88:0/2548686352 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3d7c065360 0x7f3d7c0beac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:51.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:51.825+0000 7f3d7bfff700 1 -- 172.21.15.88:0/2548686352 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d70009370 con 0x7f3d7c14d8c0 2023-10-18T16:09:51.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:51.825+0000 7f3d7bfff700 1 --2- 172.21.15.88:0/2548686352 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3d7c14d8c0 0x7f3d7c0c0360 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f3d7400c470 tx=0x7f3d74007d70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:09:51.838 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:51.825+0000 7f3d637fe700 1 -- 172.21.15.88:0/2548686352 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d7400ce70 con 0x7f3d7c14d8c0 2023-10-18T16:09:51.838 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:51.825+0000 7f3d637fe700 1 -- 172.21.15.88:0/2548686352 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d74001e90 con 0x7f3d7c14d8c0 2023-10-18T16:09:51.838 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:51.825+0000 7f3d81d67700 1 -- 172.21.15.88:0/2548686352 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d7c0bfed0 con 0x7f3d7c14d8c0 2023-10-18T16:09:51.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:51.825+0000 7f3d637fe700 1 -- 172.21.15.88:0/2548686352 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d7400f680 con 0x7f3d7c14d8c0 2023-10-18T16:09:51.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:51.825+0000 7f3d81d67700 1 -- 172.21.15.88:0/2548686352 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d7c160a50 con 0x7f3d7c14d8c0 2023-10-18T16:09:51.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:51.826+0000 7f3d81d67700 1 -- 172.21.15.88:0/2548686352 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d7c0c0060 con 0x7f3d7c14d8c0 2023-10-18T16:09:51.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:51.826+0000 7f3d637fe700 1 -- 172.21.15.88:0/2548686352 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f3d7400fa70 con 0x7f3d7c14d8c0 2023-10-18T16:09:51.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:51.827+0000 7f3d637fe700 1 --2- 172.21.15.88:0/2548686352 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f3d6805f150 0x7f3d68061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:51.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:51.827+0000 7f3d637fe700 1 -- 172.21.15.88:0/2548686352 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6725+0+0 (secure 0 0 0) 0x7f3d74079790 con 0x7f3d7c14d8c0 2023-10-18T16:09:51.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:51.827+0000 7f3d7b7fe700 1 --2- 172.21.15.88:0/2548686352 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f3d6805f150 0x7f3d68061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:51.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:51.827+0000 7f3d7b7fe700 1 --2- 172.21.15.88:0/2548686352 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f3d6805f150 0x7f3d68061610 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f3d6c0076a0 tx=0x7f3d6c009930).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:09:51.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:51.831+0000 7f3d637fe700 1 -- 172.21.15.88:0/2548686352 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f3d74047ac0 con 0x7f3d7c14d8c0 2023-10-18T16:09:52.037 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:52.034+0000 7f3d81d67700 1 -- 172.21.15.88:0/2548686352 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3d7c06c4e0 con 0x7f3d7c14d8c0 2023-10-18T16:09:52.038 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:52.034+0000 7f3d637fe700 1 -- 172.21.15.88:0/2548686352 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7f3d7404a800 con 0x7f3d7c14d8c0 2023-10-18T16:09:52.038 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:09:52.039 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:52.036+0000 7f3d81d67700 1 -- 172.21.15.88:0/2548686352 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f3d6805f150 msgr2=0x7f3d68061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:52.040 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:52.036+0000 7f3d81d67700 1 --2- 172.21.15.88:0/2548686352 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f3d6805f150 0x7f3d68061610 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f3d6c0076a0 tx=0x7f3d6c009930).stop 2023-10-18T16:09:52.040 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:52.036+0000 7f3d81d67700 1 -- 172.21.15.88:0/2548686352 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3d7c14d8c0 msgr2=0x7f3d7c0c0360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:52.040 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:52.036+0000 7f3d81d67700 1 --2- 172.21.15.88:0/2548686352 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3d7c14d8c0 0x7f3d7c0c0360 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f3d7400c470 tx=0x7f3d74007d70).stop 2023-10-18T16:09:52.040 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:52.036+0000 7f3d81d67700 1 -- 172.21.15.88:0/2548686352 shutdown_connections 2023-10-18T16:09:52.040 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:52.036+0000 7f3d81d67700 1 --2- 172.21.15.88:0/2548686352 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f3d6805f150 0x7f3d68061610 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:52.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:52.036+0000 7f3d81d67700 1 --2- 172.21.15.88:0/2548686352 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3d7c14d8c0 0x7f3d7c0c0360 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:52.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:52.036+0000 7f3d81d67700 1 --2- 172.21.15.88:0/2548686352 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3d7c065360 0x7f3d7c0beac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:52.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:52.036+0000 7f3d81d67700 1 --2- 172.21.15.88:0/2548686352 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3d7c0a9050 0x7f3d7c0ac450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:52.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:52.036+0000 7f3d81d67700 1 -- 172.21.15.88:0/2548686352 >> 172.21.15.88:0/2548686352 conn(0x7f3d7c06a810 msgr2=0x7f3d7c153ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:09:52.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:52.036+0000 7f3d81d67700 1 -- 172.21.15.88:0/2548686352 shutdown_connections 2023-10-18T16:09:52.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:52.036+0000 7f3d81d67700 1 -- 172.21.15.88:0/2548686352 wait complete. 2023-10-18T16:09:52.051 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:09:54.819 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~19s 2023-10-18T16:09:54.819 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~32s 2023-10-18T16:09:54.819 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~65s 2023-10-18T16:09:54.820 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~45s 2023-10-18T16:09:54.820 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~126s 2023-10-18T16:09:54.820 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.f is failed for ~78s 2023-10-18T16:09:55.053 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:09:55.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.174+0000 7f906ddcf700 1 Processor -- start 2023-10-18T16:09:55.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.174+0000 7f906ddcf700 1 -- start start 2023-10-18T16:09:55.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.175+0000 7f906ddcf700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9068147fa0 0x7f906814ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:55.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.175+0000 7f906ddcf700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9068062950 0x7f9068158a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:55.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.175+0000 7f906ddcf700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f906814e570 con 0x7f9068069a20 2023-10-18T16:09:55.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.175+0000 7f906ddcf700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90680bf0f0 con 0x7f9068062950 2023-10-18T16:09:55.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.175+0000 7f906ddcf700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90680bf3d0 con 0x7f9068147fa0 2023-10-18T16:09:55.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.175+0000 7f906d5ce700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9068062950 0x7f9068158a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:55.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.175+0000 7f906d5ce700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9068062950 0x7f9068158a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:53422/0 (socket says 172.21.15.88:53422) 2023-10-18T16:09:55.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.175+0000 7f906d5ce700 1 -- 172.21.15.88:0/2704334500 learned_addr learned my addr 172.21.15.88:0/2704334500 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:09:55.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.175+0000 7f9067fff700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f9068069a20 0x7f90680bcd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:47998/0 (socket says 172.21.15.88:47998) 2023-10-18T16:09:55.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.175+0000 7f906cdcd700 1 --2- 172.21.15.88:0/2704334500 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9068147fa0 0x7f906814ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:55.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.175+0000 7f906d5ce700 1 -- 172.21.15.88:0/2704334500 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9068147fa0 msgr2=0x7f906814ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:09:55.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.175+0000 7f906d5ce700 1 --2- 172.21.15.88:0/2704334500 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9068147fa0 0x7f906814ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:55.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.175+0000 7f90677fe700 1 -- 172.21.15.88:0/2704334500 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1228015511 0 0) 0x7f906814e570 con 0x7f9068069a20 2023-10-18T16:09:55.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.175+0000 7f906d5ce700 1 -- 172.21.15.88:0/2704334500 >> v1:172.21.15.88:6789/0 conn(0x7f9068069a20 legacy=0x7f90680bcd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:55.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.175+0000 7f906d5ce700 1 -- 172.21.15.88:0/2704334500 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90680bf6b0 con 0x7f9068062950 2023-10-18T16:09:55.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.175+0000 7f906d5ce700 1 --2- 172.21.15.88:0/2704334500 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9068062950 0x7f9068158a60 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f9050006fd0 tx=0x7f90500090f0).ready entity=mon.0 client_cookie=15ca91d2f52d77ff server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:09:55.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.176+0000 7f90677fe700 1 -- 172.21.15.88:0/2704334500 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f905000b600 con 0x7f9068062950 2023-10-18T16:09:55.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.176+0000 7f90677fe700 1 -- 172.21.15.88:0/2704334500 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f905000b760 con 0x7f9068062950 2023-10-18T16:09:55.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.176+0000 7f90677fe700 1 -- 172.21.15.88:0/2704334500 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f905000bae0 con 0x7f9068062950 2023-10-18T16:09:55.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.176+0000 7f906ddcf700 1 -- 172.21.15.88:0/2704334500 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9068062950 msgr2=0x7f9068158a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:55.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.176+0000 7f906ddcf700 1 --2- 172.21.15.88:0/2704334500 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9068062950 0x7f9068158a60 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f9050006fd0 tx=0x7f90500090f0).stop 2023-10-18T16:09:55.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.176+0000 7f906ddcf700 1 -- 172.21.15.88:0/2704334500 shutdown_connections 2023-10-18T16:09:55.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.176+0000 7f906ddcf700 1 --2- 172.21.15.88:0/2704334500 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9068062950 0x7f9068158a60 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:55.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.176+0000 7f906ddcf700 1 --2- 172.21.15.88:0/2704334500 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9068147fa0 0x7f906814ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:55.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.176+0000 7f906ddcf700 1 -- 172.21.15.88:0/2704334500 >> 172.21.15.88:0/2704334500 conn(0x7f9068063760 msgr2=0x7f9068063b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:09:55.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.176+0000 7f906ddcf700 1 -- 172.21.15.88:0/2704334500 shutdown_connections 2023-10-18T16:09:55.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.176+0000 7f906ddcf700 1 -- 172.21.15.88:0/2704334500 wait complete. 2023-10-18T16:09:55.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.177+0000 7f906ddcf700 1 Processor -- start 2023-10-18T16:09:55.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.177+0000 7f906ddcf700 1 -- start start 2023-10-18T16:09:55.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.178+0000 7f906ddcf700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9068062950 0x7f906814f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:55.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.178+0000 7f906ddcf700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9068069a20 0x7f90680c2c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:55.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.178+0000 7f906ddcf700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9068147fa0 0x7f90680b7520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:55.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.178+0000 7f906ddcf700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90681504c0 con 0x7f9068069a20 2023-10-18T16:09:55.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.178+0000 7f906ddcf700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9068152260 con 0x7f9068062950 2023-10-18T16:09:55.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.178+0000 7f906ddcf700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f906814f9e0 con 0x7f9068147fa0 2023-10-18T16:09:55.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.178+0000 7f9067fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9068069a20 0x7f90680c2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:55.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.178+0000 7f9067fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9068069a20 0x7f90680c2c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:53430/0 (socket says 172.21.15.88:53430) 2023-10-18T16:09:55.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.178+0000 7f9067fff700 1 -- 172.21.15.88:0/10991330 learned_addr learned my addr 172.21.15.88:0/10991330 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:09:55.191 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.178+0000 7f906cdcd700 1 --2- 172.21.15.88:0/10991330 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9068062950 0x7f906814f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:55.191 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.178+0000 7f906d5ce700 1 --2- 172.21.15.88:0/10991330 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9068147fa0 0x7f90680b7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:55.191 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.178+0000 7f9067fff700 1 -- 172.21.15.88:0/10991330 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9068147fa0 msgr2=0x7f90680b7520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:55.191 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.179+0000 7f9067fff700 1 --2- 172.21.15.88:0/10991330 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9068147fa0 0x7f90680b7520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:55.191 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.179+0000 7f9067fff700 1 -- 172.21.15.88:0/10991330 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9068062950 msgr2=0x7f906814f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:55.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.179+0000 7f9067fff700 1 --2- 172.21.15.88:0/10991330 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9068062950 0x7f906814f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:55.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.179+0000 7f9067fff700 1 -- 172.21.15.88:0/10991330 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9050009370 con 0x7f9068069a20 2023-10-18T16:09:55.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.179+0000 7f906cdcd700 1 --2- 172.21.15.88:0/10991330 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9068062950 0x7f906814f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-10-18T16:09:55.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.179+0000 7f9067fff700 1 --2- 172.21.15.88:0/10991330 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9068069a20 0x7f90680c2c20 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7f905800c810 tx=0x7f905800cbd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:09:55.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.179+0000 7f9064ff9700 1 -- 172.21.15.88:0/10991330 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9058005c70 con 0x7f9068069a20 2023-10-18T16:09:55.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.179+0000 7f9064ff9700 1 -- 172.21.15.88:0/10991330 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9058005dd0 con 0x7f9068069a20 2023-10-18T16:09:55.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.179+0000 7f906ddcf700 1 -- 172.21.15.88:0/10991330 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90681513b0 con 0x7f9068069a20 2023-10-18T16:09:55.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.179+0000 7f9064ff9700 1 -- 172.21.15.88:0/10991330 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9058007340 con 0x7f9068069a20 2023-10-18T16:09:55.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.179+0000 7f906ddcf700 1 -- 172.21.15.88:0/10991330 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9068151930 con 0x7f9068069a20 2023-10-18T16:09:55.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.180+0000 7f906ddcf700 1 -- 172.21.15.88:0/10991330 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90680671d0 con 0x7f9068069a20 2023-10-18T16:09:55.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.180+0000 7f9064ff9700 1 -- 172.21.15.88:0/10991330 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f9058007730 con 0x7f9068069a20 2023-10-18T16:09:55.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.181+0000 7f9064ff9700 1 --2- 172.21.15.88:0/10991330 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f904805f150 0x7f9048061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:55.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.181+0000 7f9064ff9700 1 -- 172.21.15.88:0/10991330 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6725+0+0 (secure 0 0 0) 0x7f905807a510 con 0x7f9068069a20 2023-10-18T16:09:55.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.181+0000 7f906cdcd700 1 --2- 172.21.15.88:0/10991330 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f904805f150 0x7f9048061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:55.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.181+0000 7f906cdcd700 1 --2- 172.21.15.88:0/10991330 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f904805f150 0x7f9048061610 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f90600098c0 tx=0x7f9060008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:09:55.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.185+0000 7f9064ff9700 1 -- 172.21.15.88:0/10991330 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f905804b580 con 0x7f9068069a20 2023-10-18T16:09:55.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.384+0000 7f906ddcf700 1 -- 172.21.15.88:0/10991330 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f9068152410 con 0x7f9068069a20 2023-10-18T16:09:55.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.384+0000 7f9064ff9700 1 -- 172.21.15.88:0/10991330 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7f905804eba0 con 0x7f9068069a20 2023-10-18T16:09:55.388 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:09:55.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.386+0000 7f906ddcf700 1 -- 172.21.15.88:0/10991330 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f904805f150 msgr2=0x7f9048061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:55.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.386+0000 7f906ddcf700 1 --2- 172.21.15.88:0/10991330 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f904805f150 0x7f9048061610 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f90600098c0 tx=0x7f9060008040).stop 2023-10-18T16:09:55.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.386+0000 7f906ddcf700 1 -- 172.21.15.88:0/10991330 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9068069a20 msgr2=0x7f90680c2c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:55.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.386+0000 7f906ddcf700 1 --2- 172.21.15.88:0/10991330 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9068069a20 0x7f90680c2c20 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7f905800c810 tx=0x7f905800cbd0).stop 2023-10-18T16:09:55.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.386+0000 7f906ddcf700 1 -- 172.21.15.88:0/10991330 shutdown_connections 2023-10-18T16:09:55.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.386+0000 7f906ddcf700 1 --2- 172.21.15.88:0/10991330 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f904805f150 0x7f9048061610 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:55.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.386+0000 7f906ddcf700 1 --2- 172.21.15.88:0/10991330 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9068147fa0 0x7f90680b7520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:55.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.386+0000 7f906ddcf700 1 --2- 172.21.15.88:0/10991330 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9068062950 0x7f906814f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:55.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.386+0000 7f906ddcf700 1 --2- 172.21.15.88:0/10991330 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9068069a20 0x7f90680c2c20 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:55.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.386+0000 7f906ddcf700 1 -- 172.21.15.88:0/10991330 >> 172.21.15.88:0/10991330 conn(0x7f9068063760 msgr2=0x7f90680bd1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:09:55.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.386+0000 7f906ddcf700 1 -- 172.21.15.88:0/10991330 shutdown_connections 2023-10-18T16:09:55.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:55.386+0000 7f906ddcf700 1 -- 172.21.15.88:0/10991330 wait complete. 2023-10-18T16:09:55.401 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:09:58.403 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:09:58.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.524+0000 7f66c6dc6700 1 Processor -- start 2023-10-18T16:09:58.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.525+0000 7f66c6dc6700 1 -- start start 2023-10-18T16:09:58.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.525+0000 7f66c6dc6700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f66c00bd8f0 0x7f66c00bed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:58.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.525+0000 7f66c6dc6700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f66c01511c0 0x7f66c00bc4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:58.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.525+0000 7f66c6dc6700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f66c00bc990 0x7f66c00be160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:58.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.525+0000 7f66c6dc6700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66c00bdcd0 con 0x7f66c00bd8f0 2023-10-18T16:09:58.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.525+0000 7f66c6dc6700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66c00bafb0 con 0x7f66c01511c0 2023-10-18T16:09:58.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.525+0000 7f66c6dc6700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66c0067d60 con 0x7f66c00bc990 2023-10-18T16:09:58.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.525+0000 7f66c4b62700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f66c00bd8f0 0x7f66c00bed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:58.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.526+0000 7f66c4b62700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f66c00bd8f0 0x7f66c00bed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46618/0 (socket says 172.21.15.88:46618) 2023-10-18T16:09:58.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.526+0000 7f66c4b62700 1 -- 172.21.15.88:0/2935378418 learned_addr learned my addr 172.21.15.88:0/2935378418 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:09:58.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.526+0000 7f66b7fff700 1 --2- 172.21.15.88:0/2935378418 >> v2:172.21.15.151:3300/0 conn(0x7f66c01511c0 0x7f66c00bc4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:58.534 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.526+0000 7f66c5363700 1 --2- 172.21.15.88:0/2935378418 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f66c00bc990 0x7f66c00be160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:58.534 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.526+0000 7f66c4b62700 1 -- 172.21.15.88:0/2935378418 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f66c00bc990 msgr2=0x7f66c00be160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:09:58.534 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.526+0000 7f66c4b62700 1 --2- 172.21.15.88:0/2935378418 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f66c00bc990 0x7f66c00be160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:58.534 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.526+0000 7f66c4b62700 1 -- 172.21.15.88:0/2935378418 >> v2:172.21.15.151:3300/0 conn(0x7f66c01511c0 msgr2=0x7f66c00bc4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:09:58.535 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.526+0000 7f66c4b62700 1 --2- 172.21.15.88:0/2935378418 >> v2:172.21.15.151:3300/0 conn(0x7f66c01511c0 0x7f66c00bc4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:58.535 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.526+0000 7f66c4b62700 1 -- 172.21.15.88:0/2935378418 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66c014dd70 con 0x7f66c00bd8f0 2023-10-18T16:09:58.535 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.526+0000 7f66b7fff700 1 --2- 172.21.15.88:0/2935378418 >> v2:172.21.15.151:3300/0 conn(0x7f66c01511c0 0x7f66c00bc4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:09:58.535 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.526+0000 7f66c4b62700 1 --2- 172.21.15.88:0/2935378418 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66c00bd8f0 0x7f66c00bed70 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f66c00699f0 tx=0x7f66bc009840).ready entity=mon.0 client_cookie=7c04e27cd9513697 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:09:58.535 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.526+0000 7f66b77fe700 1 -- 172.21.15.88:0/2935378418 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66bc00e070 con 0x7f66c00bd8f0 2023-10-18T16:09:58.536 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.526+0000 7f66b77fe700 1 -- 172.21.15.88:0/2935378418 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66bc00bae0 con 0x7f66c00bd8f0 2023-10-18T16:09:58.536 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.526+0000 7f66b77fe700 1 -- 172.21.15.88:0/2935378418 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66bc00be60 con 0x7f66c00bd8f0 2023-10-18T16:09:58.536 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.527+0000 7f66c6dc6700 1 -- 172.21.15.88:0/2935378418 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66c00bd8f0 msgr2=0x7f66c00bed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:58.536 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.527+0000 7f66c6dc6700 1 --2- 172.21.15.88:0/2935378418 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66c00bd8f0 0x7f66c00bed70 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f66c00699f0 tx=0x7f66bc009840).stop 2023-10-18T16:09:58.537 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.527+0000 7f66c6dc6700 1 -- 172.21.15.88:0/2935378418 shutdown_connections 2023-10-18T16:09:58.537 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.527+0000 7f66c6dc6700 1 --2- 172.21.15.88:0/2935378418 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f66c00bc990 0x7f66c00be160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:58.537 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.527+0000 7f66c6dc6700 1 --2- 172.21.15.88:0/2935378418 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66c00bd8f0 0x7f66c00bed70 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:58.537 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.527+0000 7f66c6dc6700 1 --2- 172.21.15.88:0/2935378418 >> v2:172.21.15.151:3300/0 conn(0x7f66c01511c0 0x7f66c00bc4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:58.537 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.527+0000 7f66c6dc6700 1 -- 172.21.15.88:0/2935378418 >> 172.21.15.88:0/2935378418 conn(0x7f66c0063760 msgr2=0x7f66c0063b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:09:58.538 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.527+0000 7f66c6dc6700 1 -- 172.21.15.88:0/2935378418 shutdown_connections 2023-10-18T16:09:58.538 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.527+0000 7f66c6dc6700 1 -- 172.21.15.88:0/2935378418 wait complete. 2023-10-18T16:09:58.538 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.528+0000 7f66c6dc6700 1 Processor -- start 2023-10-18T16:09:58.538 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.528+0000 7f66c6dc6700 1 -- start start 2023-10-18T16:09:58.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.529+0000 7f66c6dc6700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66c00bc990 0x7f66c00c4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:58.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.529+0000 7f66c6dc6700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f66c00bd8f0 0x7f66c00b9a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:58.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.529+0000 7f66c6dc6700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f66c01511c0 0x7f66c00c2c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:58.542 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.529+0000 7f66c6dc6700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66c00b2540 con 0x7f66c00bc990 2023-10-18T16:09:58.542 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.529+0000 7f66c6dc6700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66c014e5d0 con 0x7f66c00bd8f0 2023-10-18T16:09:58.542 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.529+0000 7f66c6dc6700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66c015dff0 con 0x7f66c01511c0 2023-10-18T16:09:58.542 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.529+0000 7f66c4b62700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66c00bc990 0x7f66c00c4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:58.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.529+0000 7f66c4b62700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66c00bc990 0x7f66c00c4090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46622/0 (socket says 172.21.15.88:46622) 2023-10-18T16:09:58.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.529+0000 7f66c4b62700 1 -- 172.21.15.88:0/1687978579 learned_addr learned my addr 172.21.15.88:0/1687978579 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:09:58.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.529+0000 7f66c5363700 1 --2- 172.21.15.88:0/1687978579 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f66c01511c0 0x7f66c00c2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:58.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.529+0000 7f66b7fff700 1 --2- 172.21.15.88:0/1687978579 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f66c00bd8f0 0x7f66c00b9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:58.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.529+0000 7f66c4b62700 1 -- 172.21.15.88:0/1687978579 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f66c01511c0 msgr2=0x7f66c00c2c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:58.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.529+0000 7f66c4b62700 1 --2- 172.21.15.88:0/1687978579 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f66c01511c0 0x7f66c00c2c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:58.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.529+0000 7f66c4b62700 1 -- 172.21.15.88:0/1687978579 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f66c00bd8f0 msgr2=0x7f66c00b9a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:58.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.529+0000 7f66c4b62700 1 --2- 172.21.15.88:0/1687978579 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f66c00bd8f0 0x7f66c00b9a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:58.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.529+0000 7f66c4b62700 1 -- 172.21.15.88:0/1687978579 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66bc009ae0 con 0x7f66c00bc990 2023-10-18T16:09:58.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.529+0000 7f66c4b62700 1 --2- 172.21.15.88:0/1687978579 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66c00bc990 0x7f66c00c4090 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f66bc00bab0 tx=0x7f66c00c1090).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:09:58.545 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.529+0000 7f66b4ff9700 1 -- 172.21.15.88:0/1687978579 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66bc013b20 con 0x7f66c00bc990 2023-10-18T16:09:58.545 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.529+0000 7f66b4ff9700 1 -- 172.21.15.88:0/1687978579 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66bc013c80 con 0x7f66c00bc990 2023-10-18T16:09:58.545 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.529+0000 7f66c6dc6700 1 -- 172.21.15.88:0/1687978579 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66c015d440 con 0x7f66c00bc990 2023-10-18T16:09:58.545 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.529+0000 7f66b4ff9700 1 -- 172.21.15.88:0/1687978579 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66bc01e470 con 0x7f66c00bc990 2023-10-18T16:09:58.546 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.530+0000 7f66c6dc6700 1 -- 172.21.15.88:0/1687978579 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66c015d6c0 con 0x7f66c00bc990 2023-10-18T16:09:58.546 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.530+0000 7f66c6dc6700 1 -- 172.21.15.88:0/1687978579 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66c006c4e0 con 0x7f66c00bc990 2023-10-18T16:09:58.546 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.531+0000 7f66b4ff9700 1 -- 172.21.15.88:0/1687978579 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f66bc003040 con 0x7f66c00bc990 2023-10-18T16:09:58.546 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.531+0000 7f66b4ff9700 1 --2- 172.21.15.88:0/1687978579 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f66ac05f150 0x7f66ac061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:09:58.546 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.531+0000 7f66b4ff9700 1 -- 172.21.15.88:0/1687978579 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6725+0+0 (secure 0 0 0) 0x7f66bc07b830 con 0x7f66c00bc990 2023-10-18T16:09:58.547 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.531+0000 7f66b7fff700 1 --2- 172.21.15.88:0/1687978579 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f66ac05f150 0x7f66ac061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:09:58.547 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.532+0000 7f66b7fff700 1 --2- 172.21.15.88:0/1687978579 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f66ac05f150 0x7f66ac061610 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f66c00b97b0 tx=0x7f66b0008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:09:58.547 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.535+0000 7f66b4ff9700 1 -- 172.21.15.88:0/1687978579 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f66bc04c8a0 con 0x7f66c00bc990 2023-10-18T16:09:58.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.736+0000 7f66c6dc6700 1 -- 172.21.15.88:0/1687978579 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f66c014dac0 con 0x7f66c00bc990 2023-10-18T16:09:58.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.736+0000 7f66b4ff9700 1 -- 172.21.15.88:0/1687978579 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7f66bc023020 con 0x7f66c00bc990 2023-10-18T16:09:58.740 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:09:58.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.738+0000 7f66c6dc6700 1 -- 172.21.15.88:0/1687978579 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f66ac05f150 msgr2=0x7f66ac061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:58.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.738+0000 7f66c6dc6700 1 --2- 172.21.15.88:0/1687978579 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f66ac05f150 0x7f66ac061610 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f66c00b97b0 tx=0x7f66b0008040).stop 2023-10-18T16:09:58.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.738+0000 7f66c6dc6700 1 -- 172.21.15.88:0/1687978579 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66c00bc990 msgr2=0x7f66c00c4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:09:58.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.738+0000 7f66c6dc6700 1 --2- 172.21.15.88:0/1687978579 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66c00bc990 0x7f66c00c4090 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f66bc00bab0 tx=0x7f66c00c1090).stop 2023-10-18T16:09:58.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.738+0000 7f66c6dc6700 1 -- 172.21.15.88:0/1687978579 shutdown_connections 2023-10-18T16:09:58.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.738+0000 7f66c6dc6700 1 --2- 172.21.15.88:0/1687978579 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f66ac05f150 0x7f66ac061610 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:58.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.738+0000 7f66c6dc6700 1 --2- 172.21.15.88:0/1687978579 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f66c01511c0 0x7f66c00c2c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:58.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.738+0000 7f66c6dc6700 1 --2- 172.21.15.88:0/1687978579 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66c00bc990 0x7f66c00c4090 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:58.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.738+0000 7f66c6dc6700 1 --2- 172.21.15.88:0/1687978579 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f66c00bd8f0 0x7f66c00b9a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:09:58.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.738+0000 7f66c6dc6700 1 -- 172.21.15.88:0/1687978579 >> 172.21.15.88:0/1687978579 conn(0x7f66c0063760 msgr2=0x7f66c014bba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:09:58.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.738+0000 7f66c6dc6700 1 -- 172.21.15.88:0/1687978579 shutdown_connections 2023-10-18T16:09:58.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:09:58.738+0000 7f66c6dc6700 1 -- 172.21.15.88:0/1687978579 wait complete. 2023-10-18T16:09:58.753 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:10:01.130 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~25s 2023-10-18T16:10:01.131 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~38s 2023-10-18T16:10:01.131 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~71s 2023-10-18T16:10:01.131 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~51s 2023-10-18T16:10:01.131 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~132s 2023-10-18T16:10:01.131 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.f is failed for ~84s 2023-10-18T16:10:01.754 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:10:01.879 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:01.876+0000 7f6fdf59e700 1 Processor -- start 2023-10-18T16:10:01.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:01.876+0000 7f6fdf59e700 1 -- start start 2023-10-18T16:10:01.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:01.877+0000 7f6fdf59e700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6fe00bbbe0 0x7f6fe014cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:01.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:01.877+0000 7f6fdf59e700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6fe00bef20 0x7f6fe00bf3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:01.883 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:01.877+0000 7f6fdf59e700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fe014cb00 con 0x7f6fe014fcf0 2023-10-18T16:10:01.883 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:01.877+0000 7f6fdf59e700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fe0069390 con 0x7f6fe00bbbe0 2023-10-18T16:10:01.883 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:01.877+0000 7f6fdf59e700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe00bf8a0 con 0x7f6fe00bef20 2023-10-18T16:10:01.883 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:01.877+0000 7f6fde59c700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6fe00bbbe0 0x7f6fe014cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:01.883 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:01.877+0000 7f6fddd9b700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f6fe014fcf0 0x7f6fe0068c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:41288/0 (socket says 172.21.15.88:41288) 2023-10-18T16:10:01.884 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:01.877+0000 7f6fddd9b700 1 -- 172.21.15.88:0/3276608531 learned_addr learned my addr 172.21.15.88:0/3276608531 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:10:01.884 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:01.878+0000 7f6fded9d700 1 --2- 172.21.15.88:0/3276608531 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6fe00bef20 0x7f6fe00bf3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:01.884 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:01.878+0000 7f6fde59c700 1 -- 172.21.15.88:0/3276608531 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6fe00bef20 msgr2=0x7f6fe00bf3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:10:01.884 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:01.878+0000 7f6fde59c700 1 --2- 172.21.15.88:0/3276608531 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6fe00bef20 0x7f6fe00bf3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:01.885 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:01.878+0000 7f6fde59c700 1 -- 172.21.15.88:0/3276608531 >> v1:172.21.15.88:6789/0 conn(0x7f6fe014fcf0 legacy=0x7f6fe0068c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:01.885 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:01.878+0000 7f6fde59c700 1 -- 172.21.15.88:0/3276608531 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fe01501e0 con 0x7f6fe00bbbe0 2023-10-18T16:10:01.885 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:01.878+0000 7f6fdd59a700 1 -- 172.21.15.88:0/3276608531 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2558199506 0 0) 0x7f6fe014cb00 con 0x7f6fe014fcf0 2023-10-18T16:10:01.885 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:01.878+0000 7f6fded9d700 1 --2- 172.21.15.88:0/3276608531 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6fe00bef20 0x7f6fe00bf3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-10-18T16:10:01.885 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:01.878+0000 7f6fde59c700 1 --2- 172.21.15.88:0/3276608531 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6fe00bbbe0 0x7f6fe014cf90 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f6fd000a900 tx=0x7f6fd000a200).ready entity=mon.0 client_cookie=2e95d5b6a3acdd56 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:10:01.885 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:01.878+0000 7f6fdd59a700 1 -- 172.21.15.88:0/3276608531 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd000aea0 con 0x7f6fe00bbbe0 2023-10-18T16:10:01.886 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:01.878+0000 7f6fdd59a700 1 -- 172.21.15.88:0/3276608531 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fd000c970 con 0x7f6fe00bbbe0 2023-10-18T16:10:01.886 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:01.878+0000 7f6fdd59a700 1 -- 172.21.15.88:0/3276608531 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd000ccf0 con 0x7f6fe00bbbe0 2023-10-18T16:10:01.886 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:01.879+0000 7f6fdf59e700 1 -- 172.21.15.88:0/3276608531 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6fe00bbbe0 msgr2=0x7f6fe014cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:01.886 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:01.879+0000 7f6fdf59e700 1 --2- 172.21.15.88:0/3276608531 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6fe00bbbe0 0x7f6fe014cf90 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f6fd000a900 tx=0x7f6fd000a200).stop 2023-10-18T16:10:01.887 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:01.879+0000 7f6fdf59e700 1 -- 172.21.15.88:0/3276608531 shutdown_connections 2023-10-18T16:10:01.887 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:01.879+0000 7f6fdf59e700 1 --2- 172.21.15.88:0/3276608531 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6fe00bef20 0x7f6fe00bf3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:01.887 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:01.879+0000 7f6fdf59e700 1 --2- 172.21.15.88:0/3276608531 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6fe00bbbe0 0x7f6fe014cf90 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:01.887 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:01.879+0000 7f6fdf59e700 1 -- 172.21.15.88:0/3276608531 >> 172.21.15.88:0/3276608531 conn(0x7f6fe005e050 msgr2=0x7f6fe005e460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:10:01.887 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:01.879+0000 7f6fdf59e700 1 -- 172.21.15.88:0/3276608531 shutdown_connections 2023-10-18T16:10:01.887 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:01.879+0000 7f6fdf59e700 1 -- 172.21.15.88:0/3276608531 wait complete. 2023-10-18T16:10:01.890 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:01.880+0000 7f6fdf59e700 1 Processor -- start 2023-10-18T16:10:01.891 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:01.880+0000 7f6fdf59e700 1 -- start start 2023-10-18T16:10:01.891 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:01.881+0000 7f6fdf59e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6fe00bbbe0 0x7f6fe0147f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:01.891 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:01.881+0000 7f6fdf59e700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6fe00bef20 0x7f6fe00c15a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:01.891 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:01.881+0000 7f6fdf59e700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6fe014fcf0 0x7f6fe00c1a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:01.891 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:01.881+0000 7f6fdf59e700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe00c1f80 con 0x7f6fe00bef20 2023-10-18T16:10:01.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:01.881+0000 7f6fdf59e700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe00b8d80 con 0x7f6fe00bbbe0 2023-10-18T16:10:01.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:01.881+0000 7f6fdf59e700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe0147760 con 0x7f6fe014fcf0 2023-10-18T16:10:01.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:01.881+0000 7f6fddd9b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6fe00bef20 0x7f6fe00c15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:01.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:01.881+0000 7f6fddd9b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6fe00bef20 0x7f6fe00c15a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46628/0 (socket says 172.21.15.88:46628) 2023-10-18T16:10:01.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:01.881+0000 7f6fddd9b700 1 -- 172.21.15.88:0/3614804922 learned_addr learned my addr 172.21.15.88:0/3614804922 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:10:01.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:01.881+0000 7f6fded9d700 1 --2- 172.21.15.88:0/3614804922 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6fe014fcf0 0x7f6fe00c1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:01.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:01.881+0000 7f6fde59c700 1 --2- 172.21.15.88:0/3614804922 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6fe00bbbe0 0x7f6fe0147f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:01.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:01.881+0000 7f6fddd9b700 1 -- 172.21.15.88:0/3614804922 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6fe014fcf0 msgr2=0x7f6fe00c1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:01.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:01.881+0000 7f6fddd9b700 1 --2- 172.21.15.88:0/3614804922 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6fe014fcf0 0x7f6fe00c1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:01.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:01.881+0000 7f6fddd9b700 1 -- 172.21.15.88:0/3614804922 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6fe00bbbe0 msgr2=0x7f6fe0147f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:01.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:01.881+0000 7f6fddd9b700 1 --2- 172.21.15.88:0/3614804922 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6fe00bbbe0 0x7f6fe0147f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:01.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:01.881+0000 7f6fddd9b700 1 -- 172.21.15.88:0/3614804922 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd000a440 con 0x7f6fe00bef20 2023-10-18T16:10:01.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:01.881+0000 7f6fddd9b700 1 --2- 172.21.15.88:0/3614804922 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6fe00bef20 0x7f6fe00c15a0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f6fd400c4c0 tx=0x7f6fd4007e60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:10:01.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:01.881+0000 7f6fce7fc700 1 -- 172.21.15.88:0/3614804922 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd40072e0 con 0x7f6fe00bef20 2023-10-18T16:10:01.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:01.881+0000 7f6fce7fc700 1 -- 172.21.15.88:0/3614804922 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fd4007440 con 0x7f6fe00bef20 2023-10-18T16:10:01.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:01.881+0000 7f6fdf59e700 1 -- 172.21.15.88:0/3614804922 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fe00c2460 con 0x7f6fe00bef20 2023-10-18T16:10:01.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:01.881+0000 7f6fce7fc700 1 -- 172.21.15.88:0/3614804922 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd40057a0 con 0x7f6fe00bef20 2023-10-18T16:10:01.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:01.881+0000 7f6fdf59e700 1 -- 172.21.15.88:0/3614804922 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fe015cdc0 con 0x7f6fe00bef20 2023-10-18T16:10:01.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:01.882+0000 7f6fdf59e700 1 -- 172.21.15.88:0/3614804922 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fe0067f30 con 0x7f6fe00bef20 2023-10-18T16:10:01.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:01.883+0000 7f6fce7fc700 1 -- 172.21.15.88:0/3614804922 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f6fd4005bd0 con 0x7f6fe00bef20 2023-10-18T16:10:01.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:01.883+0000 7f6fce7fc700 1 --2- 172.21.15.88:0/3614804922 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f6fc005f150 0x7f6fc0061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:01.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:01.883+0000 7f6fce7fc700 1 -- 172.21.15.88:0/3614804922 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6725+0+0 (secure 0 0 0) 0x7f6fd4079d80 con 0x7f6fe00bef20 2023-10-18T16:10:01.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:01.883+0000 7f6fde59c700 1 --2- 172.21.15.88:0/3614804922 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f6fc005f150 0x7f6fc0061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:01.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:01.884+0000 7f6fde59c700 1 --2- 172.21.15.88:0/3614804922 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f6fc005f150 0x7f6fc0061610 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f6fd000a240 tx=0x7f6fe00ba6e0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:10:01.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:01.887+0000 7f6fce7fc700 1 -- 172.21.15.88:0/3614804922 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f6fd404adf0 con 0x7f6fe00bef20 2023-10-18T16:10:02.095 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:02.091+0000 7f6fdf59e700 1 -- 172.21.15.88:0/3614804922 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f6fe01509e0 con 0x7f6fe00bef20 2023-10-18T16:10:02.095 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:02.091+0000 7f6fce7fc700 1 -- 172.21.15.88:0/3614804922 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7f6fd4018070 con 0x7f6fe00bef20 2023-10-18T16:10:02.095 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:10:02.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:02.093+0000 7f6fdf59e700 1 -- 172.21.15.88:0/3614804922 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f6fc005f150 msgr2=0x7f6fc0061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:02.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:02.093+0000 7f6fdf59e700 1 --2- 172.21.15.88:0/3614804922 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f6fc005f150 0x7f6fc0061610 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f6fd000a240 tx=0x7f6fe00ba6e0).stop 2023-10-18T16:10:02.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:02.093+0000 7f6fdf59e700 1 -- 172.21.15.88:0/3614804922 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6fe00bef20 msgr2=0x7f6fe00c15a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:02.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:02.093+0000 7f6fdf59e700 1 --2- 172.21.15.88:0/3614804922 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6fe00bef20 0x7f6fe00c15a0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f6fd400c4c0 tx=0x7f6fd4007e60).stop 2023-10-18T16:10:02.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:02.093+0000 7f6fdf59e700 1 -- 172.21.15.88:0/3614804922 shutdown_connections 2023-10-18T16:10:02.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:02.093+0000 7f6fdf59e700 1 --2- 172.21.15.88:0/3614804922 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f6fc005f150 0x7f6fc0061610 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:02.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:02.093+0000 7f6fdf59e700 1 --2- 172.21.15.88:0/3614804922 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6fe014fcf0 0x7f6fe00c1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:02.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:02.093+0000 7f6fdf59e700 1 --2- 172.21.15.88:0/3614804922 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6fe00bbbe0 0x7f6fe0147f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:02.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:02.093+0000 7f6fdf59e700 1 --2- 172.21.15.88:0/3614804922 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6fe00bef20 0x7f6fe00c15a0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:02.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:02.093+0000 7f6fdf59e700 1 -- 172.21.15.88:0/3614804922 >> 172.21.15.88:0/3614804922 conn(0x7f6fe005e050 msgr2=0x7f6fe0065fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:10:02.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:02.093+0000 7f6fdf59e700 1 -- 172.21.15.88:0/3614804922 shutdown_connections 2023-10-18T16:10:02.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:02.093+0000 7f6fdf59e700 1 -- 172.21.15.88:0/3614804922 wait complete. 2023-10-18T16:10:02.109 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:10:05.110 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:10:05.235 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.232+0000 7fe86152a700 1 Processor -- start 2023-10-18T16:10:05.235 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.232+0000 7fe86152a700 1 -- start start 2023-10-18T16:10:05.238 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.232+0000 7fe86152a700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe85c153aa0 0x7fe85c153e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:05.239 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.232+0000 7fe86152a700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fe85c0bf230 0x7fe85c154370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:05.239 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.232+0000 7fe86152a700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe85c063c70 con 0x7fe85c0654b0 2023-10-18T16:10:05.239 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.232+0000 7fe86152a700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe85c069680 con 0x7fe85c0bf230 2023-10-18T16:10:05.239 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.232+0000 7fe86152a700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe85c0bf700 con 0x7fe85c153aa0 2023-10-18T16:10:05.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.233+0000 7fe8527fc700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fe85c0654b0 0x7fe85c066b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:41304/0 (socket says 172.21.15.88:41304) 2023-10-18T16:10:05.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.233+0000 7fe8527fc700 1 -- 172.21.15.88:0/561100842 learned_addr learned my addr 172.21.15.88:0/561100842 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:10:05.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.233+0000 7fe85b7fe700 1 --2- 172.21.15.88:0/561100842 >> v2:172.21.15.151:3300/0 conn(0x7fe85c0bf230 0x7fe85c154370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:05.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.233+0000 7fe85affd700 1 --2- 172.21.15.88:0/561100842 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe85c153aa0 0x7fe85c153e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:05.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.233+0000 7fe85a7fc700 1 -- 172.21.15.88:0/561100842 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3608504392 0 0) 0x7fe85c063c70 con 0x7fe85c0654b0 2023-10-18T16:10:05.241 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.233+0000 7fe85a7fc700 1 -- 172.21.15.88:0/561100842 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8480033f0 con 0x7fe85c0654b0 2023-10-18T16:10:05.241 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.233+0000 7fe85b7fe700 1 -- 172.21.15.88:0/561100842 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe85c153aa0 msgr2=0x7fe85c153e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:10:05.241 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.233+0000 7fe85b7fe700 1 --2- 172.21.15.88:0/561100842 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe85c153aa0 0x7fe85c153e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:05.241 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.233+0000 7fe85b7fe700 1 -- 172.21.15.88:0/561100842 >> v1:172.21.15.88:6789/0 conn(0x7fe85c0654b0 legacy=0x7fe85c066b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:05.241 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.233+0000 7fe85a7fc700 1 -- 172.21.15.88:0/561100842 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2665941139 0 0) 0x7fe8480033f0 con 0x7fe85c0654b0 2023-10-18T16:10:05.242 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.233+0000 7fe85b7fe700 1 -- 172.21.15.88:0/561100842 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe85c0690c0 con 0x7fe85c0bf230 2023-10-18T16:10:05.242 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.233+0000 7fe85b7fe700 1 --2- 172.21.15.88:0/561100842 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe85c0bf230 0x7fe85c154370 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fe854009e60 tx=0x7fe854009780).ready entity=mon.1 client_cookie=78263340474695a2 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:10:05.242 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.233+0000 7fe85a7fc700 1 -- 172.21.15.88:0/561100842 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe85400e070 con 0x7fe85c0bf230 2023-10-18T16:10:05.242 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.233+0000 7fe85a7fc700 1 -- 172.21.15.88:0/561100842 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe85400b830 con 0x7fe85c0bf230 2023-10-18T16:10:05.243 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.233+0000 7fe85a7fc700 1 -- 172.21.15.88:0/561100842 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe844001470 con 0x7fe85c0bf230 2023-10-18T16:10:05.243 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.234+0000 7fe86152a700 1 -- 172.21.15.88:0/561100842 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe85c0bf230 msgr2=0x7fe85c154370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:05.243 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.234+0000 7fe86152a700 1 --2- 172.21.15.88:0/561100842 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe85c0bf230 0x7fe85c154370 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fe854009e60 tx=0x7fe854009780).stop 2023-10-18T16:10:05.243 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.234+0000 7fe86152a700 1 -- 172.21.15.88:0/561100842 shutdown_connections 2023-10-18T16:10:05.243 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.234+0000 7fe86152a700 1 --2- 172.21.15.88:0/561100842 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe85c0bf230 0x7fe85c154370 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:05.243 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.234+0000 7fe86152a700 1 --2- 172.21.15.88:0/561100842 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe85c153aa0 0x7fe85c153e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:05.244 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.234+0000 7fe86152a700 1 -- 172.21.15.88:0/561100842 >> 172.21.15.88:0/561100842 conn(0x7fe85c0c4890 msgr2=0x7fe85c0c4ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:10:05.244 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.234+0000 7fe86152a700 1 -- 172.21.15.88:0/561100842 shutdown_connections 2023-10-18T16:10:05.244 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.234+0000 7fe86152a700 1 -- 172.21.15.88:0/561100842 wait complete. 2023-10-18T16:10:05.244 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.235+0000 7fe86152a700 1 Processor -- start 2023-10-18T16:10:05.244 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.235+0000 7fe86152a700 1 -- start start 2023-10-18T16:10:05.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.236+0000 7fe86152a700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe85c0654b0 0x7fe85c14afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:05.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.236+0000 7fe86152a700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe85c0bf230 0x7fe85c160880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:05.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.236+0000 7fe86152a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe85c153aa0 0x7fe85c0c26a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:05.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.236+0000 7fe86152a700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe85c160150 con 0x7fe85c0bf230 2023-10-18T16:10:05.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.236+0000 7fe86152a700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe85c164580 con 0x7fe85c153aa0 2023-10-18T16:10:05.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.236+0000 7fe86152a700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe85c1603f0 con 0x7fe85c0654b0 2023-10-18T16:10:05.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.236+0000 7fe8527fc700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe85c0bf230 0x7fe85c160880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:05.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.236+0000 7fe8527fc700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe85c0bf230 0x7fe85c160880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46630/0 (socket says 172.21.15.88:46630) 2023-10-18T16:10:05.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.236+0000 7fe8527fc700 1 -- 172.21.15.88:0/2082377323 learned_addr learned my addr 172.21.15.88:0/2082377323 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:10:05.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.236+0000 7fe85b7fe700 1 --2- 172.21.15.88:0/2082377323 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe85c153aa0 0x7fe85c0c26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:05.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.236+0000 7fe85affd700 1 --2- 172.21.15.88:0/2082377323 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe85c0654b0 0x7fe85c14afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:05.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.236+0000 7fe8527fc700 1 -- 172.21.15.88:0/2082377323 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe85c0654b0 msgr2=0x7fe85c14afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:05.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.236+0000 7fe8527fc700 1 --2- 172.21.15.88:0/2082377323 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe85c0654b0 0x7fe85c14afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:05.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.236+0000 7fe8527fc700 1 -- 172.21.15.88:0/2082377323 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe85c153aa0 msgr2=0x7fe85c0c26a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:05.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.236+0000 7fe8527fc700 1 --2- 172.21.15.88:0/2082377323 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe85c153aa0 0x7fe85c0c26a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:05.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.236+0000 7fe8527fc700 1 -- 172.21.15.88:0/2082377323 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe854009a20 con 0x7fe85c0bf230 2023-10-18T16:10:05.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.236+0000 7fe85b7fe700 1 --2- 172.21.15.88:0/2082377323 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe85c153aa0 0x7fe85c0c26a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-10-18T16:10:05.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.236+0000 7fe85affd700 1 --2- 172.21.15.88:0/2082377323 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe85c0654b0 0x7fe85c14afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-10-18T16:10:05.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.237+0000 7fe8527fc700 1 --2- 172.21.15.88:0/2082377323 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe85c0bf230 0x7fe85c160880 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7fe84400c760 tx=0x7fe844001aa0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:10:05.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.237+0000 7fe8537fe700 1 -- 172.21.15.88:0/2082377323 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe844007320 con 0x7fe85c0bf230 2023-10-18T16:10:05.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.237+0000 7fe8537fe700 1 -- 172.21.15.88:0/2082377323 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8440074a0 con 0x7fe85c0bf230 2023-10-18T16:10:05.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.237+0000 7fe86152a700 1 -- 172.21.15.88:0/2082377323 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe85c154af0 con 0x7fe85c0bf230 2023-10-18T16:10:05.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.237+0000 7fe8537fe700 1 -- 172.21.15.88:0/2082377323 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe844005990 con 0x7fe85c0bf230 2023-10-18T16:10:05.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.237+0000 7fe86152a700 1 -- 172.21.15.88:0/2082377323 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe85c163ce0 con 0x7fe85c0bf230 2023-10-18T16:10:05.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.238+0000 7fe86152a700 1 -- 172.21.15.88:0/2082377323 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe85c160580 con 0x7fe85c0bf230 2023-10-18T16:10:05.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.238+0000 7fe8537fe700 1 -- 172.21.15.88:0/2082377323 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7fe844005d40 con 0x7fe85c0bf230 2023-10-18T16:10:05.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.238+0000 7fe8537fe700 1 --2- 172.21.15.88:0/2082377323 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fe83c05f150 0x7fe83c061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:05.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.238+0000 7fe8537fe700 1 -- 172.21.15.88:0/2082377323 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6725+0+0 (secure 0 0 0) 0x7fe84407a400 con 0x7fe85c0bf230 2023-10-18T16:10:05.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.238+0000 7fe85affd700 1 --2- 172.21.15.88:0/2082377323 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fe83c05f150 0x7fe83c061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:05.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.239+0000 7fe85affd700 1 --2- 172.21.15.88:0/2082377323 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fe83c05f150 0x7fe83c061610 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fe84c006d80 tx=0x7fe84c008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:10:05.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.242+0000 7fe8537fe700 1 -- 172.21.15.88:0/2082377323 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7fe84404b470 con 0x7fe85c0bf230 2023-10-18T16:10:05.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.445+0000 7fe86152a700 1 -- 172.21.15.88:0/2082377323 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe85c06c4e0 con 0x7fe85c0bf230 2023-10-18T16:10:05.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.446+0000 7fe8537fe700 1 -- 172.21.15.88:0/2082377323 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7fe84404ea90 con 0x7fe85c0bf230 2023-10-18T16:10:05.450 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:10:05.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.448+0000 7fe86152a700 1 -- 172.21.15.88:0/2082377323 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fe83c05f150 msgr2=0x7fe83c061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:05.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.448+0000 7fe86152a700 1 --2- 172.21.15.88:0/2082377323 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fe83c05f150 0x7fe83c061610 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fe84c006d80 tx=0x7fe84c008040).stop 2023-10-18T16:10:05.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.448+0000 7fe86152a700 1 -- 172.21.15.88:0/2082377323 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe85c0bf230 msgr2=0x7fe85c160880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:05.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.448+0000 7fe86152a700 1 --2- 172.21.15.88:0/2082377323 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe85c0bf230 0x7fe85c160880 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7fe84400c760 tx=0x7fe844001aa0).stop 2023-10-18T16:10:05.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.448+0000 7fe86152a700 1 -- 172.21.15.88:0/2082377323 shutdown_connections 2023-10-18T16:10:05.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.448+0000 7fe86152a700 1 --2- 172.21.15.88:0/2082377323 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fe83c05f150 0x7fe83c061610 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:05.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.448+0000 7fe86152a700 1 --2- 172.21.15.88:0/2082377323 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe85c153aa0 0x7fe85c0c26a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:05.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.448+0000 7fe86152a700 1 --2- 172.21.15.88:0/2082377323 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe85c0654b0 0x7fe85c14afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:05.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.448+0000 7fe86152a700 1 --2- 172.21.15.88:0/2082377323 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe85c0bf230 0x7fe85c160880 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:05.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.448+0000 7fe86152a700 1 -- 172.21.15.88:0/2082377323 >> 172.21.15.88:0/2082377323 conn(0x7fe85c0c4890 msgr2=0x7fe85c0ba310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:10:05.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.448+0000 7fe86152a700 1 -- 172.21.15.88:0/2082377323 shutdown_connections 2023-10-18T16:10:05.454 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:05.448+0000 7fe86152a700 1 -- 172.21.15.88:0/2082377323 wait complete. 2023-10-18T16:10:05.463 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:10:07.442 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~32s 2023-10-18T16:10:07.442 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~44s 2023-10-18T16:10:07.442 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~77s 2023-10-18T16:10:07.443 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~57s 2023-10-18T16:10:07.443 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~138s 2023-10-18T16:10:07.443 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.f is failed for ~91s 2023-10-18T16:10:08.464 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:10:08.491 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-18T16:10:08.509 DEBUG:teuthology.orchestra.run.smithi151:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-18T16:10:08.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.591+0000 7f75054ab700 1 Processor -- start 2023-10-18T16:10:08.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.591+0000 7f75054ab700 1 -- start start 2023-10-18T16:10:08.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.592+0000 7f75054ab700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7500063d70 0x7f75000c6e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:08.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.592+0000 7f75054ab700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f75000c2400 0x7f75000c2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:08.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.592+0000 7f75054ab700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7500151020 con 0x7f75000c6a60 2023-10-18T16:10:08.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.592+0000 7f75054ab700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7500150090 con 0x7f75000c2400 2023-10-18T16:10:08.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.592+0000 7f75054ab700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75000c2d80 con 0x7f7500063d70 2023-10-18T16:10:08.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.592+0000 7f74fffff700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f75000c6a60 0x7f7500153660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:35974/0 (socket says 172.21.15.88:35974) 2023-10-18T16:10:08.599 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.592+0000 7f74fffff700 1 -- 172.21.15.88:0/1824635160 learned_addr learned my addr 172.21.15.88:0/1824635160 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:10:08.599 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.592+0000 7f7504caa700 1 --2- 172.21.15.88:0/1824635160 >> v2:172.21.15.151:3300/0 conn(0x7f75000c2400 0x7f75000c2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:08.599 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.592+0000 7f74ff7fe700 1 --2- 172.21.15.88:0/1824635160 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7500063d70 0x7f75000c6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:08.599 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.592+0000 7f74feffd700 1 -- 172.21.15.88:0/1824635160 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 843362829 0 0) 0x7f74f0001110 con 0x7f75000c6a60 2023-10-18T16:10:08.599 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.592+0000 7f74feffd700 1 -- 172.21.15.88:0/1824635160 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74e80033f0 con 0x7f75000c6a60 2023-10-18T16:10:08.600 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.593+0000 7f74feffd700 1 -- 172.21.15.88:0/1824635160 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3596444688 0 0) 0x7f74e80033f0 con 0x7f75000c6a60 2023-10-18T16:10:08.600 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.593+0000 7f7504caa700 1 -- 172.21.15.88:0/1824635160 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7500063d70 msgr2=0x7f75000c6e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:10:08.600 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.593+0000 7f7504caa700 1 --2- 172.21.15.88:0/1824635160 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7500063d70 0x7f75000c6e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:08.600 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.593+0000 7f7504caa700 1 -- 172.21.15.88:0/1824635160 >> v1:172.21.15.88:6789/0 conn(0x7f75000c6a60 legacy=0x7f7500153660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:08.600 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.593+0000 7f7504caa700 1 -- 172.21.15.88:0/1824635160 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7500150b90 con 0x7f75000c2400 2023-10-18T16:10:08.601 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.593+0000 7f7504caa700 1 --2- 172.21.15.88:0/1824635160 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f75000c2400 0x7f75000c2890 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f74f8009e90 tx=0x7f74f8009780).ready entity=mon.1 client_cookie=1c978c9450beff68 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:10:08.601 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.593+0000 7f74feffd700 1 -- 172.21.15.88:0/1824635160 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f800e070 con 0x7f75000c2400 2023-10-18T16:10:08.601 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.593+0000 7f74feffd700 1 -- 172.21.15.88:0/1824635160 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74f800b7a0 con 0x7f75000c2400 2023-10-18T16:10:08.601 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.593+0000 7f74feffd700 1 -- 172.21.15.88:0/1824635160 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f0001600 con 0x7f75000c2400 2023-10-18T16:10:08.601 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.593+0000 7f75054ab700 1 -- 172.21.15.88:0/1824635160 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f75000c2400 msgr2=0x7f75000c2890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:08.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.593+0000 7f75054ab700 1 --2- 172.21.15.88:0/1824635160 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f75000c2400 0x7f75000c2890 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f74f8009e90 tx=0x7f74f8009780).stop 2023-10-18T16:10:08.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.593+0000 7f75054ab700 1 -- 172.21.15.88:0/1824635160 shutdown_connections 2023-10-18T16:10:08.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.593+0000 7f75054ab700 1 --2- 172.21.15.88:0/1824635160 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f75000c2400 0x7f75000c2890 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:08.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.593+0000 7f75054ab700 1 --2- 172.21.15.88:0/1824635160 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7500063d70 0x7f75000c6e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:08.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.593+0000 7f75054ab700 1 -- 172.21.15.88:0/1824635160 >> 172.21.15.88:0/1824635160 conn(0x7f750005e050 msgr2=0x7f750005e460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:10:08.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.593+0000 7f75054ab700 1 -- 172.21.15.88:0/1824635160 shutdown_connections 2023-10-18T16:10:08.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.593+0000 7f75054ab700 1 -- 172.21.15.88:0/1824635160 wait complete. 2023-10-18T16:10:08.606 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.594+0000 7f75054ab700 1 Processor -- start 2023-10-18T16:10:08.606 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.595+0000 7f75054ab700 1 -- start start 2023-10-18T16:10:08.606 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.595+0000 7f75054ab700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7500063d70 0x7f75000ba810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:08.606 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.595+0000 7f75054ab700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f75000c2400 0x7f7500154980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:08.607 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.595+0000 7f75054ab700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f75000c6a60 0x7f7500160140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:08.607 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.595+0000 7f75054ab700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7500164310 con 0x7f7500063d70 2023-10-18T16:10:08.607 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.595+0000 7f75054ab700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7500163740 con 0x7f75000c2400 2023-10-18T16:10:08.607 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.595+0000 7f75054ab700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75001639e0 con 0x7f75000c6a60 2023-10-18T16:10:08.607 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.596+0000 7f74fffff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7500063d70 0x7f75000ba810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:08.608 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.596+0000 7f74fffff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7500063d70 0x7f75000ba810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40166/0 (socket says 172.21.15.88:40166) 2023-10-18T16:10:08.608 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.596+0000 7f74fffff700 1 -- 172.21.15.88:0/867868550 learned_addr learned my addr 172.21.15.88:0/867868550 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:10:08.608 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.596+0000 7f74ff7fe700 1 --2- 172.21.15.88:0/867868550 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f75000c2400 0x7f7500154980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:08.608 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.596+0000 7f7504caa700 1 --2- 172.21.15.88:0/867868550 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f75000c6a60 0x7f7500160140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:08.609 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.596+0000 7f74fffff700 1 -- 172.21.15.88:0/867868550 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f75000c6a60 msgr2=0x7f7500160140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:08.609 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.596+0000 7f74fffff700 1 --2- 172.21.15.88:0/867868550 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f75000c6a60 0x7f7500160140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:08.609 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.596+0000 7f74fffff700 1 -- 172.21.15.88:0/867868550 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f75000c2400 msgr2=0x7f7500154980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:08.609 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.596+0000 7f74fffff700 1 --2- 172.21.15.88:0/867868550 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f75000c2400 0x7f7500154980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:08.609 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.596+0000 7f74fffff700 1 -- 172.21.15.88:0/867868550 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74f8009a20 con 0x7f7500063d70 2023-10-18T16:10:08.610 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.596+0000 7f74fffff700 1 --2- 172.21.15.88:0/867868550 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7500063d70 0x7f75000ba810 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f74f000c7e0 tx=0x7f74f0000ed0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:10:08.610 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.596+0000 7f74e7fff700 1 -- 172.21.15.88:0/867868550 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f0007170 con 0x7f7500063d70 2023-10-18T16:10:08.610 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.596+0000 7f74e7fff700 1 -- 172.21.15.88:0/867868550 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74f00072d0 con 0x7f7500063d70 2023-10-18T16:10:08.610 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.596+0000 7f75054ab700 1 -- 172.21.15.88:0/867868550 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7500163cc0 con 0x7f7500063d70 2023-10-18T16:10:08.610 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.596+0000 7f74e7fff700 1 -- 172.21.15.88:0/867868550 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f00048c0 con 0x7f7500063d70 2023-10-18T16:10:08.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.596+0000 7f75054ab700 1 -- 172.21.15.88:0/867868550 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75000bb050 con 0x7f7500063d70 2023-10-18T16:10:08.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.597+0000 7f75054ab700 1 -- 172.21.15.88:0/867868550 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7500084340 con 0x7f7500063d70 2023-10-18T16:10:08.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.597+0000 7f74e7fff700 1 -- 172.21.15.88:0/867868550 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f74f0004c70 con 0x7f7500063d70 2023-10-18T16:10:08.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.598+0000 7f74e7fff700 1 --2- 172.21.15.88:0/867868550 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f74ec05f150 0x7f74ec061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:08.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.598+0000 7f74e7fff700 1 -- 172.21.15.88:0/867868550 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6725+0+0 (secure 0 0 0) 0x7f74f0079e20 con 0x7f7500063d70 2023-10-18T16:10:08.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.598+0000 7f74ff7fe700 1 --2- 172.21.15.88:0/867868550 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f74ec05f150 0x7f74ec061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:08.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.598+0000 7f74ff7fe700 1 --2- 172.21.15.88:0/867868550 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f74ec05f150 0x7f74ec061610 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f750014eb20 tx=0x7f74f4008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:10:08.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.602+0000 7f74e7fff700 1 -- 172.21.15.88:0/867868550 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f74f004ae90 con 0x7f7500063d70 2023-10-18T16:10:08.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.803+0000 7f75054ab700 1 -- 172.21.15.88:0/867868550 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7500151020 con 0x7f7500063d70 2023-10-18T16:10:08.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.803+0000 7f74e7fff700 1 -- 172.21.15.88:0/867868550 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7f7500151020 con 0x7f7500063d70 2023-10-18T16:10:08.807 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:10:08.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.805+0000 7f75054ab700 1 -- 172.21.15.88:0/867868550 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f74ec05f150 msgr2=0x7f74ec061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:08.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.805+0000 7f75054ab700 1 --2- 172.21.15.88:0/867868550 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f74ec05f150 0x7f74ec061610 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f750014eb20 tx=0x7f74f4008040).stop 2023-10-18T16:10:08.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.805+0000 7f75054ab700 1 -- 172.21.15.88:0/867868550 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7500063d70 msgr2=0x7f75000ba810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:08.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.805+0000 7f75054ab700 1 --2- 172.21.15.88:0/867868550 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7500063d70 0x7f75000ba810 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f74f000c7e0 tx=0x7f74f0000ed0).stop 2023-10-18T16:10:08.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.805+0000 7f75054ab700 1 -- 172.21.15.88:0/867868550 shutdown_connections 2023-10-18T16:10:08.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.805+0000 7f75054ab700 1 --2- 172.21.15.88:0/867868550 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f74ec05f150 0x7f74ec061610 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:08.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.805+0000 7f75054ab700 1 --2- 172.21.15.88:0/867868550 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f75000c6a60 0x7f7500160140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:08.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.805+0000 7f75054ab700 1 --2- 172.21.15.88:0/867868550 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7500063d70 0x7f75000ba810 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:08.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.805+0000 7f75054ab700 1 --2- 172.21.15.88:0/867868550 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f75000c2400 0x7f7500154980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:08.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.805+0000 7f75054ab700 1 -- 172.21.15.88:0/867868550 >> 172.21.15.88:0/867868550 conn(0x7f750005e050 msgr2=0x7f750014df30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:10:08.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.805+0000 7f75054ab700 1 -- 172.21.15.88:0/867868550 shutdown_connections 2023-10-18T16:10:08.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:08.805+0000 7f75054ab700 1 -- 172.21.15.88:0/867868550 wait complete. 2023-10-18T16:10:08.820 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:10:11.821 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:10:11.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.941+0000 7fc60c4e2700 1 Processor -- start 2023-10-18T16:10:11.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.941+0000 7fc60c4e2700 1 -- start start 2023-10-18T16:10:11.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.942+0000 7fc60c4e2700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fc60414d8c0 0x7fc604064440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:11.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.942+0000 7fc60c4e2700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc604064150 con 0x7fc6040a9050 2023-10-18T16:10:11.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.942+0000 7fc60c4e2700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc604153a40 con 0x7fc604065360 2023-10-18T16:10:11.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.942+0000 7fc60c4e2700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc604063e70 con 0x7fc60414d8c0 2023-10-18T16:10:11.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.942+0000 7fc609a7d700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fc6040a9050 0x7fc604153ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:35978/0 (socket says 172.21.15.88:35978) 2023-10-18T16:10:11.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.942+0000 7fc609a7d700 1 -- 172.21.15.88:0/3137517512 learned_addr learned my addr 172.21.15.88:0/3137517512 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:10:11.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.942+0000 7fc60a27e700 1 --2- 172.21.15.88:0/3137517512 >> v2:172.21.15.151:3300/0 conn(0x7fc60414d8c0 0x7fc604064440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:11.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.942+0000 7fc60927c700 1 -- 172.21.15.88:0/3137517512 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1763446652 0 0) 0x7fc604064150 con 0x7fc6040a9050 2023-10-18T16:10:11.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.942+0000 7fc60927c700 1 -- 172.21.15.88:0/3137517512 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5ec0033f0 con 0x7fc6040a9050 2023-10-18T16:10:11.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.942+0000 7fc60927c700 1 -- 172.21.15.88:0/3137517512 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3486655765 0 0) 0x7fc5ec0033f0 con 0x7fc6040a9050 2023-10-18T16:10:11.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.942+0000 7fc60927c700 1 -- 172.21.15.88:0/3137517512 >> v2:172.21.15.151:3300/0 conn(0x7fc60414d8c0 msgr2=0x7fc604064440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:10:11.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.942+0000 7fc60927c700 1 --2- 172.21.15.88:0/3137517512 >> v2:172.21.15.151:3300/0 conn(0x7fc60414d8c0 0x7fc604064440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:11.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.942+0000 7fc60927c700 1 -- 172.21.15.88:0/3137517512 >> v1:172.21.15.151:6789/0 conn(0x7fc604065360 legacy=0x7fc60415acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:11.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.942+0000 7fc60927c700 1 -- 172.21.15.88:0/3137517512 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6040c72c0 con 0x7fc6040a9050 2023-10-18T16:10:11.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.942+0000 7fc60927c700 1 -- 172.21.15.88:0/3137517512 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441715075 0 0) 0x7fc5f8001470 con 0x7fc6040a9050 2023-10-18T16:10:11.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.942+0000 7fc60927c700 1 -- 172.21.15.88:0/3137517512 >> v1:172.21.15.88:6789/0 conn(0x7fc6040a9050 legacy=0x7fc604153ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:11.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.942+0000 7fc60927c700 1 --2- 172.21.15.88:0/3137517512 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5ec0038e0 0x7fc5ec003d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:11.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.942+0000 7fc60927c700 1 --2- 172.21.15.88:0/3137517512 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc5ec004690 0x7fc5ec004b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:11.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.942+0000 7fc60927c700 1 --2- 172.21.15.88:0/3137517512 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc5ec005050 0x7fc5ec00b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:11.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.942+0000 7fc60927c700 1 -- 172.21.15.88:0/3137517512 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc604063e70 con 0x7fc5ec0038e0 2023-10-18T16:10:11.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.942+0000 7fc60927c700 1 -- 172.21.15.88:0/3137517512 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5ec00bfe0 con 0x7fc5ec004690 2023-10-18T16:10:11.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.942+0000 7fc60927c700 1 -- 172.21.15.88:0/3137517512 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5ec00c1a0 con 0x7fc5ec005050 2023-10-18T16:10:11.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.943+0000 7fc60a27e700 1 --2- 172.21.15.88:0/3137517512 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5ec0038e0 0x7fc5ec003d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:11.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.943+0000 7fc609a7d700 1 --2- 172.21.15.88:0/3137517512 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc5ec004690 0x7fc5ec004b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:11.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.943+0000 7fc60aa7f700 1 --2- 172.21.15.88:0/3137517512 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc5ec005050 0x7fc5ec00b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:11.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.943+0000 7fc60a27e700 1 -- 172.21.15.88:0/3137517512 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc5ec005050 msgr2=0x7fc5ec00b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:10:11.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.943+0000 7fc60a27e700 1 --2- 172.21.15.88:0/3137517512 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc5ec005050 0x7fc5ec00b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:11.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.943+0000 7fc60a27e700 1 -- 172.21.15.88:0/3137517512 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc5ec004690 msgr2=0x7fc5ec004b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:10:11.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.943+0000 7fc60a27e700 1 --2- 172.21.15.88:0/3137517512 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc5ec004690 0x7fc5ec004b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:11.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.943+0000 7fc60a27e700 1 -- 172.21.15.88:0/3137517512 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5ec0033f0 con 0x7fc5ec0038e0 2023-10-18T16:10:11.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.943+0000 7fc60aa7f700 1 --2- 172.21.15.88:0/3137517512 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc5ec005050 0x7fc5ec00b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-10-18T16:10:11.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.943+0000 7fc609a7d700 1 --2- 172.21.15.88:0/3137517512 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc5ec004690 0x7fc5ec004b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:10:11.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.943+0000 7fc60aa7f700 1 -- 172.21.15.88:0/3137517512 reap_dead start 2023-10-18T16:10:11.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.943+0000 7fc60a27e700 1 --2- 172.21.15.88:0/3137517512 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5ec0038e0 0x7fc5ec003d00 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7fc5fc0094d0 tx=0x7fc5fc00b040).ready entity=mon.0 client_cookie=ca0d3f8d2f5ce580 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:10:11.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.943+0000 7fc60927c700 1 -- 172.21.15.88:0/3137517512 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5fc009ac0 con 0x7fc5ec0038e0 2023-10-18T16:10:11.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.943+0000 7fc60927c700 1 -- 172.21.15.88:0/3137517512 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5fc00f040 con 0x7fc5ec0038e0 2023-10-18T16:10:11.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.943+0000 7fc60927c700 1 -- 172.21.15.88:0/3137517512 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5fc00eca0 con 0x7fc5ec0038e0 2023-10-18T16:10:11.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.943+0000 7fc60c4e2700 1 -- 172.21.15.88:0/3137517512 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5ec0038e0 msgr2=0x7fc5ec003d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:11.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.943+0000 7fc60c4e2700 1 --2- 172.21.15.88:0/3137517512 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5ec0038e0 0x7fc5ec003d00 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7fc5fc0094d0 tx=0x7fc5fc00b040).stop 2023-10-18T16:10:11.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.944+0000 7fc60c4e2700 1 -- 172.21.15.88:0/3137517512 shutdown_connections 2023-10-18T16:10:11.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.944+0000 7fc60c4e2700 1 --2- 172.21.15.88:0/3137517512 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5ec0038e0 0x7fc5ec003d00 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:11.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.944+0000 7fc60c4e2700 1 -- 172.21.15.88:0/3137517512 >> 172.21.15.88:0/3137517512 conn(0x7fc60406a810 msgr2=0x7fc6040c6a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:10:11.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.944+0000 7fc60c4e2700 1 -- 172.21.15.88:0/3137517512 shutdown_connections 2023-10-18T16:10:11.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.944+0000 7fc60c4e2700 1 -- 172.21.15.88:0/3137517512 wait complete. 2023-10-18T16:10:11.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.945+0000 7fc60c4e2700 1 Processor -- start 2023-10-18T16:10:11.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.945+0000 7fc60c4e2700 1 -- start start 2023-10-18T16:10:11.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.945+0000 7fc60c4e2700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc5ec0038e0 0x7fc6040aa010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:11.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.945+0000 7fc60c4e2700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc604153db0 0x7fc604163db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:11.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.945+0000 7fc60c4e2700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc604154230 0x7fc6041642a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:11.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.945+0000 7fc60c4e2700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc604164790 con 0x7fc604154230 2023-10-18T16:10:11.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.945+0000 7fc60c4e2700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc604163920 con 0x7fc604153db0 2023-10-18T16:10:11.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.945+0000 7fc60c4e2700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc60414e1d0 con 0x7fc5ec0038e0 2023-10-18T16:10:11.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.945+0000 7fc60aa7f700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc604154230 0x7fc6041642a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:11.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.946+0000 7fc60aa7f700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc604154230 0x7fc6041642a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40176/0 (socket says 172.21.15.88:40176) 2023-10-18T16:10:11.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.946+0000 7fc60aa7f700 1 -- 172.21.15.88:0/1452271671 learned_addr learned my addr 172.21.15.88:0/1452271671 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:10:11.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.946+0000 7fc609a7d700 1 --2- 172.21.15.88:0/1452271671 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc604153db0 0x7fc604163db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:11.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.946+0000 7fc60a27e700 1 --2- 172.21.15.88:0/1452271671 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc5ec0038e0 0x7fc6040aa010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:11.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.946+0000 7fc60aa7f700 1 -- 172.21.15.88:0/1452271671 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc5ec0038e0 msgr2=0x7fc6040aa010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:11.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.946+0000 7fc60aa7f700 1 --2- 172.21.15.88:0/1452271671 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc5ec0038e0 0x7fc6040aa010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:11.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.946+0000 7fc60aa7f700 1 -- 172.21.15.88:0/1452271671 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc604153db0 msgr2=0x7fc604163db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:11.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.946+0000 7fc60aa7f700 1 --2- 172.21.15.88:0/1452271671 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc604153db0 0x7fc604163db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:11.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.946+0000 7fc60aa7f700 1 -- 172.21.15.88:0/1452271671 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc60414e4b0 con 0x7fc604154230 2023-10-18T16:10:11.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.946+0000 7fc60a27e700 1 --2- 172.21.15.88:0/1452271671 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc5ec0038e0 0x7fc6040aa010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-10-18T16:10:11.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.946+0000 7fc609a7d700 1 --2- 172.21.15.88:0/1452271671 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc604153db0 0x7fc604163db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-10-18T16:10:11.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.946+0000 7fc60aa7f700 1 --2- 172.21.15.88:0/1452271671 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc604154230 0x7fc6041642a0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7fc60405f630 tx=0x7fc5f4009040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:10:11.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.946+0000 7fc6027fc700 1 -- 172.21.15.88:0/1452271671 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5f400dd90 con 0x7fc604154230 2023-10-18T16:10:11.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.946+0000 7fc6027fc700 1 -- 172.21.15.88:0/1452271671 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5f4013420 con 0x7fc604154230 2023-10-18T16:10:11.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.946+0000 7fc60c4e2700 1 -- 172.21.15.88:0/1452271671 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc60414e790 con 0x7fc604154230 2023-10-18T16:10:11.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.946+0000 7fc6027fc700 1 -- 172.21.15.88:0/1452271671 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5f40137a0 con 0x7fc604154230 2023-10-18T16:10:11.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.946+0000 7fc60c4e2700 1 -- 172.21.15.88:0/1452271671 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6040aa500 con 0x7fc604154230 2023-10-18T16:10:11.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.947+0000 7fc60c4e2700 1 -- 172.21.15.88:0/1452271671 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc604163ad0 con 0x7fc604154230 2023-10-18T16:10:11.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.948+0000 7fc6027fc700 1 -- 172.21.15.88:0/1452271671 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7fc5f4013b50 con 0x7fc604154230 2023-10-18T16:10:11.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.948+0000 7fc6027fc700 1 --2- 172.21.15.88:0/1452271671 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fc5f005f150 0x7fc5f0061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:11.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.948+0000 7fc6027fc700 1 -- 172.21.15.88:0/1452271671 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6725+0+0 (secure 0 0 0) 0x7fc5f4079550 con 0x7fc604154230 2023-10-18T16:10:11.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.948+0000 7fc60a27e700 1 --2- 172.21.15.88:0/1452271671 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fc5f005f150 0x7fc5f0061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:11.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.948+0000 7fc60a27e700 1 --2- 172.21.15.88:0/1452271671 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fc5f005f150 0x7fc5f0061610 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fc5fc00f010 tx=0x7fc5fc01a000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:10:11.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:11.952+0000 7fc6027fc700 1 -- 172.21.15.88:0/1452271671 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7fc604163ad0 con 0x7fc604154230 2023-10-18T16:10:12.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:12.150+0000 7fc60c4e2700 1 -- 172.21.15.88:0/1452271671 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc604163ad0 con 0x7fc604154230 2023-10-18T16:10:12.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:12.151+0000 7fc6027fc700 1 -- 172.21.15.88:0/1452271671 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7fc604163ad0 con 0x7fc604154230 2023-10-18T16:10:12.155 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:10:12.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:12.152+0000 7fc60c4e2700 1 -- 172.21.15.88:0/1452271671 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fc5f005f150 msgr2=0x7fc5f0061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:12.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:12.152+0000 7fc60c4e2700 1 --2- 172.21.15.88:0/1452271671 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fc5f005f150 0x7fc5f0061610 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fc5fc00f010 tx=0x7fc5fc01a000).stop 2023-10-18T16:10:12.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:12.152+0000 7fc60c4e2700 1 -- 172.21.15.88:0/1452271671 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc604154230 msgr2=0x7fc6041642a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:12.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:12.152+0000 7fc60c4e2700 1 --2- 172.21.15.88:0/1452271671 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc604154230 0x7fc6041642a0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7fc60405f630 tx=0x7fc5f4009040).stop 2023-10-18T16:10:12.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:12.152+0000 7fc60c4e2700 1 -- 172.21.15.88:0/1452271671 shutdown_connections 2023-10-18T16:10:12.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:12.152+0000 7fc60c4e2700 1 --2- 172.21.15.88:0/1452271671 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fc5f005f150 0x7fc5f0061610 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:12.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:12.152+0000 7fc60c4e2700 1 --2- 172.21.15.88:0/1452271671 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc604154230 0x7fc6041642a0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:12.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:12.152+0000 7fc60c4e2700 1 --2- 172.21.15.88:0/1452271671 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc5ec0038e0 0x7fc6040aa010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:12.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:12.152+0000 7fc60c4e2700 1 --2- 172.21.15.88:0/1452271671 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc604153db0 0x7fc604163db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:12.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:12.152+0000 7fc60c4e2700 1 -- 172.21.15.88:0/1452271671 >> 172.21.15.88:0/1452271671 conn(0x7fc60406a810 msgr2=0x7fc6040c6720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:10:12.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:12.153+0000 7fc60c4e2700 1 -- 172.21.15.88:0/1452271671 shutdown_connections 2023-10-18T16:10:12.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:12.153+0000 7fc60c4e2700 1 -- 172.21.15.88:0/1452271671 wait complete. 2023-10-18T16:10:12.168 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:10:13.753 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~38s 2023-10-18T16:10:13.754 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~51s 2023-10-18T16:10:13.754 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~84s 2023-10-18T16:10:13.754 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~64s 2023-10-18T16:10:13.755 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~145s 2023-10-18T16:10:13.755 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.f is failed for ~97s 2023-10-18T16:10:15.169 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:10:15.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.291+0000 7f8de6ac9700 1 Processor -- start 2023-10-18T16:10:15.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.291+0000 7f8de6ac9700 1 -- start start 2023-10-18T16:10:15.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.291+0000 7f8de6ac9700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f8de014d8c0 0x7f8de0064440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:15.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.291+0000 7f8de6ac9700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8de0065360 0x7f8de0064930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:15.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.291+0000 7f8de6ac9700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8de0064150 con 0x7f8de00a9050 2023-10-18T16:10:15.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.291+0000 7f8de6ac9700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8de0153a40 con 0x7f8de0065360 2023-10-18T16:10:15.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.291+0000 7f8de6ac9700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8de0063e70 con 0x7f8de014d8c0 2023-10-18T16:10:15.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.292+0000 7f8de5066700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8de0065360 0x7f8de0064930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:15.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.292+0000 7f8de5066700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8de0065360 0x7f8de0064930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40180/0 (socket says 172.21.15.88:40180) 2023-10-18T16:10:15.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.292+0000 7f8de5066700 1 -- 172.21.15.88:0/3596388919 learned_addr learned my addr 172.21.15.88:0/3596388919 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:10:15.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.292+0000 7f8de4865700 1 --2- 172.21.15.88:0/3596388919 >> v2:172.21.15.151:3300/0 conn(0x7f8de014d8c0 0x7f8de0064440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:15.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.292+0000 7f8de5066700 1 -- 172.21.15.88:0/3596388919 >> v2:172.21.15.151:3300/0 conn(0x7f8de014d8c0 msgr2=0x7f8de0064440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:10:15.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.292+0000 7f8de5066700 1 --2- 172.21.15.88:0/3596388919 >> v2:172.21.15.151:3300/0 conn(0x7f8de014d8c0 0x7f8de0064440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:15.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.292+0000 7f8de5066700 1 -- 172.21.15.88:0/3596388919 >> v1:172.21.15.151:6789/0 conn(0x7f8de00a9050 legacy=0x7f8de0153ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:15.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.292+0000 7f8de5066700 1 -- 172.21.15.88:0/3596388919 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8de00c72c0 con 0x7f8de0065360 2023-10-18T16:10:15.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.292+0000 7f8de5066700 1 --2- 172.21.15.88:0/3596388919 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8de0065360 0x7f8de0064930 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f8de0062d50 tx=0x7f8dd80090f0).ready entity=mon.0 client_cookie=8dc877a5112e61ac server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:10:15.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.292+0000 7f8dd77fe700 1 -- 172.21.15.88:0/3596388919 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd800b600 con 0x7f8de0065360 2023-10-18T16:10:15.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.292+0000 7f8dd77fe700 1 -- 172.21.15.88:0/3596388919 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dd800b760 con 0x7f8de0065360 2023-10-18T16:10:15.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.292+0000 7f8dd77fe700 1 -- 172.21.15.88:0/3596388919 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd800bb00 con 0x7f8de0065360 2023-10-18T16:10:15.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.293+0000 7f8de6ac9700 1 -- 172.21.15.88:0/3596388919 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8de0065360 msgr2=0x7f8de0064930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:15.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.293+0000 7f8de6ac9700 1 --2- 172.21.15.88:0/3596388919 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8de0065360 0x7f8de0064930 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f8de0062d50 tx=0x7f8dd80090f0).stop 2023-10-18T16:10:15.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.293+0000 7f8de6ac9700 1 -- 172.21.15.88:0/3596388919 shutdown_connections 2023-10-18T16:10:15.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.293+0000 7f8de6ac9700 1 --2- 172.21.15.88:0/3596388919 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8de0065360 0x7f8de0064930 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:15.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.293+0000 7f8de6ac9700 1 --2- 172.21.15.88:0/3596388919 >> v2:172.21.15.151:3300/0 conn(0x7f8de014d8c0 0x7f8de0064440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:15.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.293+0000 7f8de6ac9700 1 -- 172.21.15.88:0/3596388919 >> 172.21.15.88:0/3596388919 conn(0x7f8de006a810 msgr2=0x7f8de00c6a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:10:15.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.293+0000 7f8de6ac9700 1 -- 172.21.15.88:0/3596388919 shutdown_connections 2023-10-18T16:10:15.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.293+0000 7f8de6ac9700 1 -- 172.21.15.88:0/3596388919 wait complete. 2023-10-18T16:10:15.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.294+0000 7f8de6ac9700 1 Processor -- start 2023-10-18T16:10:15.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.294+0000 7f8de6ac9700 1 -- start start 2023-10-18T16:10:15.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.295+0000 7f8de6ac9700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8de0065360 0x7f8de014f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:15.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.295+0000 7f8de6ac9700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8de00a9050 0x7f8de0164170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:15.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.295+0000 7f8de6ac9700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8de014d8c0 0x7f8de0164660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:15.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.295+0000 7f8de6ac9700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8de00bb4f0 con 0x7f8de00a9050 2023-10-18T16:10:15.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.295+0000 7f8de6ac9700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8de00bbd90 con 0x7f8de0065360 2023-10-18T16:10:15.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.295+0000 7f8de6ac9700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8de00bd7d0 con 0x7f8de014d8c0 2023-10-18T16:10:15.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.295+0000 7f8dd7fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8de00a9050 0x7f8de0164170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:15.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.295+0000 7f8dd7fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8de00a9050 0x7f8de0164170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40186/0 (socket says 172.21.15.88:40186) 2023-10-18T16:10:15.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.295+0000 7f8dd7fff700 1 -- 172.21.15.88:0/3463518086 learned_addr learned my addr 172.21.15.88:0/3463518086 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:10:15.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.295+0000 7f8de4865700 1 --2- 172.21.15.88:0/3463518086 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8de0065360 0x7f8de014f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:15.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.295+0000 7f8de5066700 1 --2- 172.21.15.88:0/3463518086 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8de014d8c0 0x7f8de0164660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:15.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.295+0000 7f8dd7fff700 1 -- 172.21.15.88:0/3463518086 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8de014d8c0 msgr2=0x7f8de0164660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:15.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.295+0000 7f8dd7fff700 1 --2- 172.21.15.88:0/3463518086 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8de014d8c0 0x7f8de0164660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:15.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.295+0000 7f8dd7fff700 1 -- 172.21.15.88:0/3463518086 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8de0065360 msgr2=0x7f8de014f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:15.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.295+0000 7f8dd7fff700 1 --2- 172.21.15.88:0/3463518086 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8de0065360 0x7f8de014f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:15.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.295+0000 7f8dd7fff700 1 -- 172.21.15.88:0/3463518086 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dd8009370 con 0x7f8de00a9050 2023-10-18T16:10:15.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.295+0000 7f8de5066700 1 --2- 172.21.15.88:0/3463518086 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8de014d8c0 0x7f8de0164660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:10:15.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.295+0000 7f8dd7fff700 1 --2- 172.21.15.88:0/3463518086 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8de00a9050 0x7f8de0164170 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f8dd000c440 tx=0x7f8dd0007dc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:10:15.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.295+0000 7f8dd4ff9700 1 -- 172.21.15.88:0/3463518086 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd000cea0 con 0x7f8de00a9050 2023-10-18T16:10:15.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.295+0000 7f8dd4ff9700 1 -- 172.21.15.88:0/3463518086 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dd000f470 con 0x7f8de00a9050 2023-10-18T16:10:15.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.295+0000 7f8de6ac9700 1 -- 172.21.15.88:0/3463518086 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8de0164d10 con 0x7f8de00a9050 2023-10-18T16:10:15.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.295+0000 7f8dd4ff9700 1 -- 172.21.15.88:0/3463518086 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd000f810 con 0x7f8de00a9050 2023-10-18T16:10:15.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.295+0000 7f8de6ac9700 1 -- 172.21.15.88:0/3463518086 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8de00bba20 con 0x7f8de00a9050 2023-10-18T16:10:15.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.297+0000 7f8de6ac9700 1 -- 172.21.15.88:0/3463518086 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8de006c4e0 con 0x7f8de00a9050 2023-10-18T16:10:15.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.297+0000 7f8dd4ff9700 1 -- 172.21.15.88:0/3463518086 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f8dd000fc40 con 0x7f8de00a9050 2023-10-18T16:10:15.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.297+0000 7f8dd4ff9700 1 --2- 172.21.15.88:0/3463518086 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f8dcc05f150 0x7f8dcc061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:15.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.297+0000 7f8dd4ff9700 1 -- 172.21.15.88:0/3463518086 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6725+0+0 (secure 0 0 0) 0x7f8dd0079780 con 0x7f8de00a9050 2023-10-18T16:10:15.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.297+0000 7f8de4865700 1 --2- 172.21.15.88:0/3463518086 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f8dcc05f150 0x7f8dcc061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:15.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.298+0000 7f8de4865700 1 --2- 172.21.15.88:0/3463518086 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f8dcc05f150 0x7f8dcc061610 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f8ddc0075a0 tx=0x7f8ddc0097b0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:10:15.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.301+0000 7f8dd4ff9700 1 -- 172.21.15.88:0/3463518086 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f8dd0048ac0 con 0x7f8de00a9050 2023-10-18T16:10:15.505 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.502+0000 7f8de6ac9700 1 -- 172.21.15.88:0/3463518086 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f8de00c75a0 con 0x7f8de00a9050 2023-10-18T16:10:15.506 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.502+0000 7f8dd4ff9700 1 -- 172.21.15.88:0/3463518086 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7f8dd0017020 con 0x7f8de00a9050 2023-10-18T16:10:15.506 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:10:15.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.504+0000 7f8de6ac9700 1 -- 172.21.15.88:0/3463518086 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f8dcc05f150 msgr2=0x7f8dcc061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:15.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.504+0000 7f8de6ac9700 1 --2- 172.21.15.88:0/3463518086 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f8dcc05f150 0x7f8dcc061610 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f8ddc0075a0 tx=0x7f8ddc0097b0).stop 2023-10-18T16:10:15.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.504+0000 7f8de6ac9700 1 -- 172.21.15.88:0/3463518086 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8de00a9050 msgr2=0x7f8de0164170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:15.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.504+0000 7f8de6ac9700 1 --2- 172.21.15.88:0/3463518086 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8de00a9050 0x7f8de0164170 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f8dd000c440 tx=0x7f8dd0007dc0).stop 2023-10-18T16:10:15.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.504+0000 7f8de6ac9700 1 -- 172.21.15.88:0/3463518086 shutdown_connections 2023-10-18T16:10:15.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.504+0000 7f8de6ac9700 1 --2- 172.21.15.88:0/3463518086 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f8dcc05f150 0x7f8dcc061610 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:15.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.504+0000 7f8de6ac9700 1 --2- 172.21.15.88:0/3463518086 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8de014d8c0 0x7f8de0164660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:15.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.504+0000 7f8de6ac9700 1 --2- 172.21.15.88:0/3463518086 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8de0065360 0x7f8de014f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:15.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.504+0000 7f8de6ac9700 1 --2- 172.21.15.88:0/3463518086 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8de00a9050 0x7f8de0164170 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:15.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.504+0000 7f8de6ac9700 1 -- 172.21.15.88:0/3463518086 >> 172.21.15.88:0/3463518086 conn(0x7f8de006a810 msgr2=0x7f8de0153ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:10:15.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.504+0000 7f8de6ac9700 1 -- 172.21.15.88:0/3463518086 shutdown_connections 2023-10-18T16:10:15.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:15.504+0000 7f8de6ac9700 1 -- 172.21.15.88:0/3463518086 wait complete. 2023-10-18T16:10:15.519 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:10:18.520 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:10:18.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.640+0000 7fc3a4786700 1 Processor -- start 2023-10-18T16:10:18.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.640+0000 7fc3a4786700 1 -- start start 2023-10-18T16:10:18.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.641+0000 7fc3a4786700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc39c0a9050 0x7fc39c153ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:18.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.641+0000 7fc3a4786700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc39c064150 con 0x7fc39c14d8c0 2023-10-18T16:10:18.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.641+0000 7fc3a4786700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc39c153a40 con 0x7fc39c065360 2023-10-18T16:10:18.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.641+0000 7fc3a4786700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc39c063e70 con 0x7fc39c0a9050 2023-10-18T16:10:18.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.641+0000 7fc3a2522700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fc39c14d8c0 0x7fc39c064440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:52002/0 (socket says 172.21.15.88:52002) 2023-10-18T16:10:18.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.641+0000 7fc3a2522700 1 -- 172.21.15.88:0/354822171 learned_addr learned my addr 172.21.15.88:0/354822171 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:10:18.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.641+0000 7fc3a1520700 1 -- 172.21.15.88:0/354822171 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2863275791 0 0) 0x7fc39c064150 con 0x7fc39c14d8c0 2023-10-18T16:10:18.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.641+0000 7fc3a1520700 1 -- 172.21.15.88:0/354822171 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3880033f0 con 0x7fc39c14d8c0 2023-10-18T16:10:18.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.641+0000 7fc3a1d21700 1 --2- 172.21.15.88:0/354822171 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc39c0a9050 0x7fc39c153ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:18.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.641+0000 7fc3a1520700 1 -- 172.21.15.88:0/354822171 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2845353064 0 0) 0x7fc3880033f0 con 0x7fc39c14d8c0 2023-10-18T16:10:18.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.641+0000 7fc3a1520700 1 -- 172.21.15.88:0/354822171 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc39c0a9050 msgr2=0x7fc39c153ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:10:18.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.641+0000 7fc3a1520700 1 --2- 172.21.15.88:0/354822171 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc39c0a9050 0x7fc39c153ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:18.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.641+0000 7fc3a1520700 1 -- 172.21.15.88:0/354822171 >> v1:172.21.15.151:6789/0 conn(0x7fc39c065360 legacy=0x7fc39c15acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:18.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.641+0000 7fc3a1520700 1 -- 172.21.15.88:0/354822171 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc39c0c72c0 con 0x7fc39c14d8c0 2023-10-18T16:10:18.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.641+0000 7fc3a1520700 1 -- 172.21.15.88:0/354822171 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441715075 0 0) 0x7fc398001470 con 0x7fc39c14d8c0 2023-10-18T16:10:18.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.641+0000 7fc3a1520700 1 -- 172.21.15.88:0/354822171 >> v1:172.21.15.88:6789/0 conn(0x7fc39c14d8c0 legacy=0x7fc39c064440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:18.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.641+0000 7fc3a1520700 1 --2- 172.21.15.88:0/354822171 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc39c0a9050 0x7fc388003910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:18.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.642+0000 7fc3a1520700 1 --2- 172.21.15.88:0/354822171 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc388003e40 0x7fc388004690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:18.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.642+0000 7fc3a1520700 1 --2- 172.21.15.88:0/354822171 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc388004b80 0x7fc388005000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:18.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.642+0000 7fc3a1520700 1 -- 172.21.15.88:0/354822171 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc39c063e70 con 0x7fc388003e40 2023-10-18T16:10:18.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.642+0000 7fc3a1520700 1 -- 172.21.15.88:0/354822171 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc38800ba10 con 0x7fc388004b80 2023-10-18T16:10:18.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.642+0000 7fc3a1520700 1 -- 172.21.15.88:0/354822171 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc38800bbd0 con 0x7fc39c0a9050 2023-10-18T16:10:18.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.642+0000 7fc3a1d21700 1 --2- 172.21.15.88:0/354822171 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc388003e40 0x7fc388004690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:18.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.642+0000 7fc3a2522700 1 --2- 172.21.15.88:0/354822171 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc39c0a9050 0x7fc388003910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:18.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.642+0000 7fc3a2d23700 1 --2- 172.21.15.88:0/354822171 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc388004b80 0x7fc388005000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:18.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.642+0000 7fc3a1d21700 1 -- 172.21.15.88:0/354822171 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc39c0a9050 msgr2=0x7fc388003910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:10:18.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.642+0000 7fc3a1d21700 1 --2- 172.21.15.88:0/354822171 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc39c0a9050 0x7fc388003910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:18.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.642+0000 7fc3a1d21700 1 -- 172.21.15.88:0/354822171 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc388004b80 msgr2=0x7fc388005000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:10:18.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.642+0000 7fc3a1d21700 1 --2- 172.21.15.88:0/354822171 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc388004b80 0x7fc388005000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:18.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.642+0000 7fc3a1d21700 1 -- 172.21.15.88:0/354822171 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3880033f0 con 0x7fc388003e40 2023-10-18T16:10:18.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.642+0000 7fc3a1d21700 1 --2- 172.21.15.88:0/354822171 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc388003e40 0x7fc388004690 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7fc38c009a80 tx=0x7fc38c009060).ready entity=mon.0 client_cookie=e919d6ba9a5d42a6 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:10:18.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.642+0000 7fc3a1520700 1 -- 172.21.15.88:0/354822171 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc38c009430 con 0x7fc388003e40 2023-10-18T16:10:18.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.642+0000 7fc3a1520700 1 -- 172.21.15.88:0/354822171 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc38c00e640 con 0x7fc388003e40 2023-10-18T16:10:18.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.642+0000 7fc3a1520700 1 -- 172.21.15.88:0/354822171 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc38c00e9c0 con 0x7fc388003e40 2023-10-18T16:10:18.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.643+0000 7fc3a4786700 1 -- 172.21.15.88:0/354822171 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc388003e40 msgr2=0x7fc388004690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:18.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.643+0000 7fc3a4786700 1 --2- 172.21.15.88:0/354822171 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc388003e40 0x7fc388004690 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7fc38c009a80 tx=0x7fc38c009060).stop 2023-10-18T16:10:18.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.643+0000 7fc3a2d23700 1 -- 172.21.15.88:0/354822171 reap_dead start 2023-10-18T16:10:18.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.643+0000 7fc3a4786700 1 -- 172.21.15.88:0/354822171 shutdown_connections 2023-10-18T16:10:18.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.643+0000 7fc3a4786700 1 -- 172.21.15.88:0/354822171 >> 172.21.15.88:0/354822171 conn(0x7fc39c06a810 msgr2=0x7fc39c0c6a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:10:18.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.643+0000 7fc3a4786700 1 -- 172.21.15.88:0/354822171 shutdown_connections 2023-10-18T16:10:18.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.643+0000 7fc3a4786700 1 -- 172.21.15.88:0/354822171 wait complete. 2023-10-18T16:10:18.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.644+0000 7fc3a4786700 1 Processor -- start 2023-10-18T16:10:18.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.644+0000 7fc3a4786700 1 -- start start 2023-10-18T16:10:18.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.645+0000 7fc3a4786700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc388003e40 0x7fc39c0ba890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:18.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.645+0000 7fc3a4786700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc39c1493a0 0x7fc39c0bf680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:18.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.645+0000 7fc3a4786700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc39c160500 0x7fc39c160980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:18.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.645+0000 7fc3a4786700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc39c0bdb20 con 0x7fc388003e40 2023-10-18T16:10:18.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.645+0000 7fc3a4786700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc39c0ba420 con 0x7fc39c1493a0 2023-10-18T16:10:18.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.645+0000 7fc3a4786700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc39c0bad80 con 0x7fc39c160500 2023-10-18T16:10:18.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.645+0000 7fc3a2522700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc388003e40 0x7fc39c0ba890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:18.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.645+0000 7fc3a2522700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc388003e40 0x7fc39c0ba890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55000/0 (socket says 172.21.15.88:55000) 2023-10-18T16:10:18.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.645+0000 7fc3a2522700 1 -- 172.21.15.88:0/182086984 learned_addr learned my addr 172.21.15.88:0/182086984 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:10:18.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.645+0000 7fc3a1d21700 1 --2- 172.21.15.88:0/182086984 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc39c1493a0 0x7fc39c0bf680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:18.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.645+0000 7fc3a2d23700 1 --2- 172.21.15.88:0/182086984 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc39c160500 0x7fc39c160980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:18.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.645+0000 7fc3a2522700 1 -- 172.21.15.88:0/182086984 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc39c160500 msgr2=0x7fc39c160980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:18.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.645+0000 7fc3a2522700 1 --2- 172.21.15.88:0/182086984 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc39c160500 0x7fc39c160980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:18.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.645+0000 7fc3a2522700 1 -- 172.21.15.88:0/182086984 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc39c1493a0 msgr2=0x7fc39c0bf680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:18.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.645+0000 7fc3a2522700 1 --2- 172.21.15.88:0/182086984 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc39c1493a0 0x7fc39c0bf680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:18.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.645+0000 7fc3a2522700 1 -- 172.21.15.88:0/182086984 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc39c0be030 con 0x7fc388003e40 2023-10-18T16:10:18.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.645+0000 7fc3a2522700 1 --2- 172.21.15.88:0/182086984 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc388003e40 0x7fc39c0ba890 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7fc3980049e0 tx=0x7fc398001e50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:10:18.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.645+0000 7fc3927fc700 1 -- 172.21.15.88:0/182086984 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc39800f660 con 0x7fc388003e40 2023-10-18T16:10:18.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.645+0000 7fc3927fc700 1 -- 172.21.15.88:0/182086984 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc39800f7c0 con 0x7fc388003e40 2023-10-18T16:10:18.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.645+0000 7fc3927fc700 1 -- 172.21.15.88:0/182086984 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc39800fb60 con 0x7fc388003e40 2023-10-18T16:10:18.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.645+0000 7fc3a4786700 1 -- 172.21.15.88:0/182086984 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc39c0bdd50 con 0x7fc388003e40 2023-10-18T16:10:18.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.645+0000 7fc3a4786700 1 -- 172.21.15.88:0/182086984 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc39c0ab770 con 0x7fc388003e40 2023-10-18T16:10:18.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.646+0000 7fc3a4786700 1 -- 172.21.15.88:0/182086984 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc39c06c4e0 con 0x7fc388003e40 2023-10-18T16:10:18.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.647+0000 7fc3927fc700 1 -- 172.21.15.88:0/182086984 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7fc39801c360 con 0x7fc388003e40 2023-10-18T16:10:18.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.647+0000 7fc3927fc700 1 --2- 172.21.15.88:0/182086984 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fc38005f120 0x7fc3800615e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:18.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.647+0000 7fc3927fc700 1 -- 172.21.15.88:0/182086984 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6725+0+0 (secure 0 0 0) 0x7fc398079b20 con 0x7fc388003e40 2023-10-18T16:10:18.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.647+0000 7fc3a1d21700 1 --2- 172.21.15.88:0/182086984 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fc38005f120 0x7fc3800615e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:18.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.647+0000 7fc3a1d21700 1 --2- 172.21.15.88:0/182086984 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fc38005f120 0x7fc3800615e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fc38c006d50 tx=0x7fc38c01a000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:10:18.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.651+0000 7fc3927fc700 1 -- 172.21.15.88:0/182086984 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7fc39804ab30 con 0x7fc388003e40 2023-10-18T16:10:18.855 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.851+0000 7fc3a4786700 1 -- 172.21.15.88:0/182086984 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc39c0c75a0 con 0x7fc388003e40 2023-10-18T16:10:18.855 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.852+0000 7fc3927fc700 1 -- 172.21.15.88:0/182086984 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7fc39c0c75a0 con 0x7fc388003e40 2023-10-18T16:10:18.856 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:10:18.857 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.853+0000 7fc3a4786700 1 -- 172.21.15.88:0/182086984 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fc38005f120 msgr2=0x7fc3800615e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:18.857 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.853+0000 7fc3a4786700 1 --2- 172.21.15.88:0/182086984 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fc38005f120 0x7fc3800615e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fc38c006d50 tx=0x7fc38c01a000).stop 2023-10-18T16:10:18.857 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.853+0000 7fc3a4786700 1 -- 172.21.15.88:0/182086984 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc388003e40 msgr2=0x7fc39c0ba890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:18.858 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.853+0000 7fc3a4786700 1 --2- 172.21.15.88:0/182086984 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc388003e40 0x7fc39c0ba890 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7fc3980049e0 tx=0x7fc398001e50).stop 2023-10-18T16:10:18.858 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.853+0000 7fc3a4786700 1 -- 172.21.15.88:0/182086984 shutdown_connections 2023-10-18T16:10:18.858 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.853+0000 7fc3a4786700 1 --2- 172.21.15.88:0/182086984 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fc38005f120 0x7fc3800615e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:18.858 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.853+0000 7fc3a4786700 1 --2- 172.21.15.88:0/182086984 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc39c160500 0x7fc39c160980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:18.858 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.853+0000 7fc3a4786700 1 --2- 172.21.15.88:0/182086984 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc388003e40 0x7fc39c0ba890 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:18.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.853+0000 7fc3a4786700 1 --2- 172.21.15.88:0/182086984 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc39c1493a0 0x7fc39c0bf680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:18.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.853+0000 7fc3a4786700 1 -- 172.21.15.88:0/182086984 >> 172.21.15.88:0/182086984 conn(0x7fc39c06a810 msgr2=0x7fc39c153d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:10:18.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.853+0000 7fc3a4786700 1 -- 172.21.15.88:0/182086984 shutdown_connections 2023-10-18T16:10:18.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:18.853+0000 7fc3a4786700 1 -- 172.21.15.88:0/182086984 wait complete. 2023-10-18T16:10:18.868 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:10:20.065 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~44s 2023-10-18T16:10:20.066 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~57s 2023-10-18T16:10:20.066 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~90s 2023-10-18T16:10:20.066 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~70s 2023-10-18T16:10:20.066 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~151s 2023-10-18T16:10:20.066 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.f is failed for ~103s 2023-10-18T16:10:21.870 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:10:21.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:21.989+0000 7f407f4c6700 1 Processor -- start 2023-10-18T16:10:21.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:21.989+0000 7f407f4c6700 1 -- start start 2023-10-18T16:10:21.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:21.990+0000 7f407f4c6700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f4078151910 0x7f4078068650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:21.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:21.990+0000 7f407f4c6700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40780c5660 0x7f4078151cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:21.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:21.990+0000 7f407f4c6700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4078068b40 0x7f40781521e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:21.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:21.990+0000 7f407f4c6700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4078067b20 con 0x7f4078068b40 2023-10-18T16:10:21.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:21.990+0000 7f407f4c6700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4078067dd0 con 0x7f4078151910 2023-10-18T16:10:21.997 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:21.990+0000 7f407f4c6700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40780b9b60 con 0x7f40780c5660 2023-10-18T16:10:21.997 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:21.990+0000 7f407da63700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4078068b40 0x7f40781521e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:21.997 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:21.991+0000 7f407da63700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4078068b40 0x7f40781521e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55002/0 (socket says 172.21.15.88:55002) 2023-10-18T16:10:21.997 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:21.991+0000 7f407da63700 1 -- 172.21.15.88:0/2284487857 learned_addr learned my addr 172.21.15.88:0/2284487857 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:10:21.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:21.991+0000 7f407d262700 1 --2- 172.21.15.88:0/2284487857 >> v2:172.21.15.151:3300/0 conn(0x7f4078151910 0x7f4078068650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:21.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:21.991+0000 7f407ca61700 1 --2- 172.21.15.88:0/2284487857 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40780c5660 0x7f4078151cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:21.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:21.991+0000 7f407da63700 1 -- 172.21.15.88:0/2284487857 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40780c5660 msgr2=0x7f4078151cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:10:21.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:21.991+0000 7f407da63700 1 --2- 172.21.15.88:0/2284487857 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40780c5660 0x7f4078151cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:21.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:21.991+0000 7f407da63700 1 -- 172.21.15.88:0/2284487857 >> v2:172.21.15.151:3300/0 conn(0x7f4078151910 msgr2=0x7f4078068650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:10:21.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:21.991+0000 7f407da63700 1 --2- 172.21.15.88:0/2284487857 >> v2:172.21.15.151:3300/0 conn(0x7f4078151910 0x7f4078068650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:21.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:21.991+0000 7f407da63700 1 -- 172.21.15.88:0/2284487857 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40780b9ff0 con 0x7f4078068b40 2023-10-18T16:10:21.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:21.991+0000 7f407da63700 1 --2- 172.21.15.88:0/2284487857 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4078068b40 0x7f40781521e0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f40780c55f0 tx=0x7f4070009840).ready entity=mon.0 client_cookie=c0235ec323ba82f1 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:10:21.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:21.991+0000 7f406ffff700 1 -- 172.21.15.88:0/2284487857 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f407000e070 con 0x7f4078068b40 2023-10-18T16:10:22.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:21.991+0000 7f406ffff700 1 -- 172.21.15.88:0/2284487857 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f407000ba00 con 0x7f4078068b40 2023-10-18T16:10:22.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:21.991+0000 7f406ffff700 1 -- 172.21.15.88:0/2284487857 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f407000bd80 con 0x7f4078068b40 2023-10-18T16:10:22.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:21.992+0000 7f407f4c6700 1 -- 172.21.15.88:0/2284487857 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4078068b40 msgr2=0x7f40781521e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:22.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:21.992+0000 7f407f4c6700 1 --2- 172.21.15.88:0/2284487857 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4078068b40 0x7f40781521e0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f40780c55f0 tx=0x7f4070009840).stop 2023-10-18T16:10:22.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:21.992+0000 7f407f4c6700 1 -- 172.21.15.88:0/2284487857 shutdown_connections 2023-10-18T16:10:22.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:21.992+0000 7f407f4c6700 1 --2- 172.21.15.88:0/2284487857 >> v2:172.21.15.151:3300/0 conn(0x7f4078151910 0x7f4078068650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:22.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:21.992+0000 7f407f4c6700 1 --2- 172.21.15.88:0/2284487857 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4078068b40 0x7f40781521e0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:22.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:21.992+0000 7f407f4c6700 1 --2- 172.21.15.88:0/2284487857 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40780c5660 0x7f4078151cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:22.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:21.992+0000 7f407f4c6700 1 -- 172.21.15.88:0/2284487857 >> 172.21.15.88:0/2284487857 conn(0x7f40780c4940 msgr2=0x7f40780c4d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:10:22.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:21.992+0000 7f407f4c6700 1 -- 172.21.15.88:0/2284487857 shutdown_connections 2023-10-18T16:10:22.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:21.992+0000 7f407f4c6700 1 -- 172.21.15.88:0/2284487857 wait complete. 2023-10-18T16:10:22.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:21.993+0000 7f407f4c6700 1 Processor -- start 2023-10-18T16:10:22.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:21.993+0000 7f407f4c6700 1 -- start start 2023-10-18T16:10:22.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:21.994+0000 7f407f4c6700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4078068b40 0x7f4078146590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:22.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:21.994+0000 7f407f4c6700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40780c5660 0x7f40781533b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:22.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:21.994+0000 7f407f4c6700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4078151910 0x7f40781538a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:22.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:21.994+0000 7f407f4c6700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4078149250 con 0x7f40780c5660 2023-10-18T16:10:22.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:21.994+0000 7f407f4c6700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4078152f20 con 0x7f4078151910 2023-10-18T16:10:22.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:21.994+0000 7f407f4c6700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4078153d90 con 0x7f4078068b40 2023-10-18T16:10:22.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:21.994+0000 7f407ca61700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40780c5660 0x7f40781533b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:22.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:21.994+0000 7f407ca61700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40780c5660 0x7f40781533b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55012/0 (socket says 172.21.15.88:55012) 2023-10-18T16:10:22.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:21.994+0000 7f407ca61700 1 -- 172.21.15.88:0/2039870545 learned_addr learned my addr 172.21.15.88:0/2039870545 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:10:22.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:21.994+0000 7f407d262700 1 --2- 172.21.15.88:0/2039870545 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4078068b40 0x7f4078146590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:22.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:21.994+0000 7f407da63700 1 --2- 172.21.15.88:0/2039870545 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4078151910 0x7f40781538a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:22.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:21.994+0000 7f407ca61700 1 -- 172.21.15.88:0/2039870545 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4078068b40 msgr2=0x7f4078146590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:22.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:21.994+0000 7f407ca61700 1 --2- 172.21.15.88:0/2039870545 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4078068b40 0x7f4078146590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:22.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:21.994+0000 7f407ca61700 1 -- 172.21.15.88:0/2039870545 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4078151910 msgr2=0x7f40781538a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:22.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:21.994+0000 7f407ca61700 1 --2- 172.21.15.88:0/2039870545 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4078151910 0x7f40781538a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:22.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:21.994+0000 7f407ca61700 1 -- 172.21.15.88:0/2039870545 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4070009ae0 con 0x7f40780c5660 2023-10-18T16:10:22.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:21.994+0000 7f407ca61700 1 --2- 172.21.15.88:0/2039870545 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40780c5660 0x7f40781533b0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f406800e950 tx=0x7f4068009060).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:10:22.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:21.995+0000 7f406d7fa700 1 -- 172.21.15.88:0/2039870545 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4068012600 con 0x7f40780c5660 2023-10-18T16:10:22.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:21.995+0000 7f406d7fa700 1 -- 172.21.15.88:0/2039870545 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4068012760 con 0x7f40780c5660 2023-10-18T16:10:22.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:21.995+0000 7f406d7fa700 1 -- 172.21.15.88:0/2039870545 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4068012ae0 con 0x7f40780c5660 2023-10-18T16:10:22.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:21.995+0000 7f407f4c6700 1 -- 172.21.15.88:0/2039870545 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4078148c90 con 0x7f40780c5660 2023-10-18T16:10:22.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:21.995+0000 7f407f4c6700 1 -- 172.21.15.88:0/2039870545 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40781528d0 con 0x7f40780c5660 2023-10-18T16:10:22.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:21.996+0000 7f407f4c6700 1 -- 172.21.15.88:0/2039870545 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4078152730 con 0x7f40780c5660 2023-10-18T16:10:22.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:21.996+0000 7f406d7fa700 1 -- 172.21.15.88:0/2039870545 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f406801b400 con 0x7f40780c5660 2023-10-18T16:10:22.010 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:21.996+0000 7f406d7fa700 1 --2- 172.21.15.88:0/2039870545 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f405c05f150 0x7f405c061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:22.010 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:21.996+0000 7f406d7fa700 1 -- 172.21.15.88:0/2039870545 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6725+0+0 (secure 0 0 0) 0x7f406807a8f0 con 0x7f40780c5660 2023-10-18T16:10:22.010 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:21.997+0000 7f407d262700 1 --2- 172.21.15.88:0/2039870545 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f405c05f150 0x7f405c061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:22.010 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:21.997+0000 7f407d262700 1 --2- 172.21.15.88:0/2039870545 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f405c05f150 0x7f405c061610 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f4074009b90 tx=0x7f4074009060).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:10:22.011 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:22.000+0000 7f406d7fa700 1 -- 172.21.15.88:0/2039870545 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f4078152730 con 0x7f40780c5660 2023-10-18T16:10:22.204 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:22.200+0000 7f407f4c6700 1 -- 172.21.15.88:0/2039870545 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4078152730 con 0x7f40780c5660 2023-10-18T16:10:22.204 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:22.201+0000 7f406d7fa700 1 -- 172.21.15.88:0/2039870545 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7f40680160e0 con 0x7f40780c5660 2023-10-18T16:10:22.205 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:10:22.206 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:22.202+0000 7f407f4c6700 1 -- 172.21.15.88:0/2039870545 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f405c05f150 msgr2=0x7f405c061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:22.207 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:22.202+0000 7f407f4c6700 1 --2- 172.21.15.88:0/2039870545 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f405c05f150 0x7f405c061610 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f4074009b90 tx=0x7f4074009060).stop 2023-10-18T16:10:22.207 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:22.202+0000 7f407f4c6700 1 -- 172.21.15.88:0/2039870545 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40780c5660 msgr2=0x7f40781533b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:22.207 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:22.202+0000 7f407f4c6700 1 --2- 172.21.15.88:0/2039870545 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40780c5660 0x7f40781533b0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f406800e950 tx=0x7f4068009060).stop 2023-10-18T16:10:22.207 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:22.203+0000 7f407f4c6700 1 -- 172.21.15.88:0/2039870545 shutdown_connections 2023-10-18T16:10:22.207 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:22.203+0000 7f407f4c6700 1 --2- 172.21.15.88:0/2039870545 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f405c05f150 0x7f405c061610 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:22.208 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:22.203+0000 7f407f4c6700 1 --2- 172.21.15.88:0/2039870545 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4078151910 0x7f40781538a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:22.208 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:22.203+0000 7f407f4c6700 1 --2- 172.21.15.88:0/2039870545 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4078068b40 0x7f4078146590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:22.208 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:22.203+0000 7f407f4c6700 1 --2- 172.21.15.88:0/2039870545 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40780c5660 0x7f40781533b0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:22.208 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:22.203+0000 7f407f4c6700 1 -- 172.21.15.88:0/2039870545 >> 172.21.15.88:0/2039870545 conn(0x7f40780c4940 msgr2=0x7f4078149840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:10:22.208 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:22.203+0000 7f407f4c6700 1 -- 172.21.15.88:0/2039870545 shutdown_connections 2023-10-18T16:10:22.209 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:22.203+0000 7f407f4c6700 1 -- 172.21.15.88:0/2039870545 wait complete. 2023-10-18T16:10:22.218 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:10:25.218 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:10:25.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.338+0000 7f1bfe8b8700 1 Processor -- start 2023-10-18T16:10:25.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.339+0000 7f1bfe8b8700 1 -- start start 2023-10-18T16:10:25.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.339+0000 7f1bfe8b8700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1bf80bfb30 0x7f1bf80bff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:25.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.339+0000 7f1bfe8b8700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f1bf80c0610 0x7f1bf80bc790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:25.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.339+0000 7f1bfe8b8700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bf80b6650 con 0x7f1bf80bcc80 2023-10-18T16:10:25.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.339+0000 7f1bfe8b8700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bf814d650 con 0x7f1bf80c0610 2023-10-18T16:10:25.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.339+0000 7f1bfe8b8700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf805e0f0 con 0x7f1bf80bfb30 2023-10-18T16:10:25.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.340+0000 7f1bf7fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1bf80bfb30 0x7f1bf80bff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:25.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.340+0000 7f1bf77fe700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f1bf80c0610 0x7f1bf80bc790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:25.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.340+0000 7f1bf77fe700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f1bf80c0610 0x7f1bf80bc790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.88:49558/0 (socket says 172.21.15.88:49558) 2023-10-18T16:10:25.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.340+0000 7f1bf7fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1bf80bfb30 0x7f1bf80bff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.88:48084/0 (socket says 172.21.15.88:48084) 2023-10-18T16:10:25.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.340+0000 7f1bf77fe700 1 -- 172.21.15.88:0/2938958594 learned_addr learned my addr 172.21.15.88:0/2938958594 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:10:25.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.340+0000 7f1bfce55700 1 --1- 172.21.15.88:0/2938958594 >> v1:172.21.15.151:6789/0 conn(0x7f1bf80bcc80 0x7f1bf814dae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.88:52094/0 (socket says 172.21.15.88:52094) 2023-10-18T16:10:25.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.340+0000 7f1bf77fe700 1 -- 172.21.15.88:0/2938958594 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1bf80bfb30 msgr2=0x7f1bf80bff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:10:25.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.340+0000 7f1bf77fe700 1 --2- 172.21.15.88:0/2938958594 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1bf80bfb30 0x7f1bf80bff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:25.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.340+0000 7f1bf77fe700 1 -- 172.21.15.88:0/2938958594 >> v1:172.21.15.151:6789/0 conn(0x7f1bf80bcc80 legacy=0x7f1bf814dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:25.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.340+0000 7f1bf77fe700 1 -- 172.21.15.88:0/2938958594 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bf805e390 con 0x7f1bf80c0610 2023-10-18T16:10:25.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.340+0000 7f1bf7fff700 1 --2- 172.21.15.88:0/2938958594 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1bf80bfb30 0x7f1bf80bff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2023-10-18T16:10:25.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.340+0000 7f1bf77fe700 1 --2- 172.21.15.88:0/2938958594 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1bf80c0610 0x7f1bf80bc790 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f1bec006fd0 tx=0x7f1bec0090f0).ready entity=mon.1 client_cookie=eba4e9ff016ce48f server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:10:25.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.340+0000 7f1bf6ffd700 1 -- 172.21.15.88:0/2938958594 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bec00b580 con 0x7f1bf80c0610 2023-10-18T16:10:25.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.340+0000 7f1bf6ffd700 1 -- 172.21.15.88:0/2938958594 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bec00b6e0 con 0x7f1bf80c0610 2023-10-18T16:10:25.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.340+0000 7f1bf6ffd700 1 -- 172.21.15.88:0/2938958594 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bec00ba60 con 0x7f1bf80c0610 2023-10-18T16:10:25.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.341+0000 7f1bfe8b8700 1 -- 172.21.15.88:0/2938958594 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1bf80c0610 msgr2=0x7f1bf80bc790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:25.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.341+0000 7f1bfe8b8700 1 --2- 172.21.15.88:0/2938958594 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1bf80c0610 0x7f1bf80bc790 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f1bec006fd0 tx=0x7f1bec0090f0).stop 2023-10-18T16:10:25.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.341+0000 7f1bfe8b8700 1 -- 172.21.15.88:0/2938958594 shutdown_connections 2023-10-18T16:10:25.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.341+0000 7f1bfe8b8700 1 --2- 172.21.15.88:0/2938958594 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1bf80bfb30 0x7f1bf80bff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:25.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.341+0000 7f1bfe8b8700 1 --2- 172.21.15.88:0/2938958594 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1bf80c0610 0x7f1bf80bc790 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:25.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.341+0000 7f1bfe8b8700 1 -- 172.21.15.88:0/2938958594 >> 172.21.15.88:0/2938958594 conn(0x7f1bf8066f70 msgr2=0x7f1bf8067380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:10:25.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.341+0000 7f1bfe8b8700 1 -- 172.21.15.88:0/2938958594 shutdown_connections 2023-10-18T16:10:25.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.341+0000 7f1bfe8b8700 1 -- 172.21.15.88:0/2938958594 wait complete. 2023-10-18T16:10:25.353 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.342+0000 7f1bfe8b8700 1 Processor -- start 2023-10-18T16:10:25.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.342+0000 7f1bfe8b8700 1 -- start start 2023-10-18T16:10:25.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.343+0000 7f1bfe8b8700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1bf80bcc80 0x7f1bf80b8970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:25.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.343+0000 7f1bfe8b8700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1bf80bfb30 0x7f1bf814a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:25.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.343+0000 7f1bfe8b8700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1bf80c0610 0x7f1bf814ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:25.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.343+0000 7f1bfe8b8700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf814b880 con 0x7f1bf80bcc80 2023-10-18T16:10:25.355 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.343+0000 7f1bfe8b8700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf80b84e0 con 0x7f1bf80c0610 2023-10-18T16:10:25.355 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.343+0000 7f1bfe8b8700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf8147010 con 0x7f1bf80bfb30 2023-10-18T16:10:25.355 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.343+0000 7f1bf7fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1bf80bcc80 0x7f1bf80b8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:25.355 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.343+0000 7f1bf7fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1bf80bcc80 0x7f1bf80b8970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55028/0 (socket says 172.21.15.88:55028) 2023-10-18T16:10:25.356 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.343+0000 7f1bf7fff700 1 -- 172.21.15.88:0/676050528 learned_addr learned my addr 172.21.15.88:0/676050528 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:10:25.356 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.343+0000 7f1bf77fe700 1 --2- 172.21.15.88:0/676050528 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1bf80bfb30 0x7f1bf814a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:25.356 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.343+0000 7f1bfce55700 1 --2- 172.21.15.88:0/676050528 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1bf80c0610 0x7f1bf814ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:25.356 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.343+0000 7f1bf7fff700 1 -- 172.21.15.88:0/676050528 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1bf80bfb30 msgr2=0x7f1bf814a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:25.356 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.343+0000 7f1bf7fff700 1 --2- 172.21.15.88:0/676050528 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1bf80bfb30 0x7f1bf814a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:25.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.343+0000 7f1bf7fff700 1 -- 172.21.15.88:0/676050528 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1bf80c0610 msgr2=0x7f1bf814ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:25.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.343+0000 7f1bf7fff700 1 --2- 172.21.15.88:0/676050528 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1bf80c0610 0x7f1bf814ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:25.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.343+0000 7f1bf7fff700 1 -- 172.21.15.88:0/676050528 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1be80092a0 con 0x7f1bf80bcc80 2023-10-18T16:10:25.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.343+0000 7f1bf7fff700 1 --2- 172.21.15.88:0/676050528 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1bf80bcc80 0x7f1bf80b8970 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7f1be800ea50 tx=0x7f1be8009060).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:10:25.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.344+0000 7f1bdffff700 1 -- 172.21.15.88:0/676050528 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1be800ee90 con 0x7f1bf80bcc80 2023-10-18T16:10:25.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.344+0000 7f1bdffff700 1 -- 172.21.15.88:0/676050528 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1be800c850 con 0x7f1bf80bcc80 2023-10-18T16:10:25.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.344+0000 7f1bfe8b8700 1 -- 172.21.15.88:0/676050528 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bec009370 con 0x7f1bf80bcc80 2023-10-18T16:10:25.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.344+0000 7f1bdffff700 1 -- 172.21.15.88:0/676050528 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1be800cbd0 con 0x7f1bf80bcc80 2023-10-18T16:10:25.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.344+0000 7f1bfe8b8700 1 -- 172.21.15.88:0/676050528 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bf80b8170 con 0x7f1bf80bcc80 2023-10-18T16:10:25.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.345+0000 7f1bfe8b8700 1 -- 172.21.15.88:0/676050528 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bf80b87c0 con 0x7f1bf80bcc80 2023-10-18T16:10:25.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.345+0000 7f1bdffff700 1 -- 172.21.15.88:0/676050528 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f1be801b400 con 0x7f1bf80bcc80 2023-10-18T16:10:25.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.345+0000 7f1bdffff700 1 --2- 172.21.15.88:0/676050528 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f1be405f120 0x7f1be40615e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:25.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.345+0000 7f1bdffff700 1 -- 172.21.15.88:0/676050528 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6725+0+0 (secure 0 0 0) 0x7f1be807a970 con 0x7f1bf80bcc80 2023-10-18T16:10:25.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.345+0000 7f1bf77fe700 1 --2- 172.21.15.88:0/676050528 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f1be405f120 0x7f1be40615e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:25.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.346+0000 7f1bf77fe700 1 --2- 172.21.15.88:0/676050528 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f1be405f120 0x7f1be40615e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f1bec006fd0 tx=0x7f1bec017000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:10:25.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.349+0000 7f1bdffff700 1 -- 172.21.15.88:0/676050528 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f1be804b980 con 0x7f1bf80bcc80 2023-10-18T16:10:25.553 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.550+0000 7f1bfe8b8700 1 -- 172.21.15.88:0/676050528 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f1bf806c4e0 con 0x7f1bf80bcc80 2023-10-18T16:10:25.554 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.550+0000 7f1bdffff700 1 -- 172.21.15.88:0/676050528 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7f1be804efa0 con 0x7f1bf80bcc80 2023-10-18T16:10:25.554 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:10:25.556 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.552+0000 7f1bfe8b8700 1 -- 172.21.15.88:0/676050528 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f1be405f120 msgr2=0x7f1be40615e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:25.556 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.552+0000 7f1bfe8b8700 1 --2- 172.21.15.88:0/676050528 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f1be405f120 0x7f1be40615e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f1bec006fd0 tx=0x7f1bec017000).stop 2023-10-18T16:10:25.556 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.552+0000 7f1bfe8b8700 1 -- 172.21.15.88:0/676050528 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1bf80bcc80 msgr2=0x7f1bf80b8970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:25.556 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.552+0000 7f1bfe8b8700 1 --2- 172.21.15.88:0/676050528 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1bf80bcc80 0x7f1bf80b8970 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7f1be800ea50 tx=0x7f1be8009060).stop 2023-10-18T16:10:25.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.552+0000 7f1bfe8b8700 1 -- 172.21.15.88:0/676050528 shutdown_connections 2023-10-18T16:10:25.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.552+0000 7f1bfe8b8700 1 --2- 172.21.15.88:0/676050528 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f1be405f120 0x7f1be40615e0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:25.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.552+0000 7f1bfe8b8700 1 --2- 172.21.15.88:0/676050528 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1bf80c0610 0x7f1bf814ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:25.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.552+0000 7f1bfe8b8700 1 --2- 172.21.15.88:0/676050528 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1bf80bcc80 0x7f1bf80b8970 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:25.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.552+0000 7f1bfe8b8700 1 --2- 172.21.15.88:0/676050528 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1bf80bfb30 0x7f1bf814a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:25.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.552+0000 7f1bfe8b8700 1 -- 172.21.15.88:0/676050528 >> 172.21.15.88:0/676050528 conn(0x7f1bf8066f70 msgr2=0x7f1bf80626e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:10:25.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.552+0000 7f1bfe8b8700 1 -- 172.21.15.88:0/676050528 shutdown_connections 2023-10-18T16:10:25.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:25.552+0000 7f1bfe8b8700 1 -- 172.21.15.88:0/676050528 wait complete. 2023-10-18T16:10:25.567 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:10:26.376 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~50s 2023-10-18T16:10:26.376 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~63s 2023-10-18T16:10:26.377 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~96s 2023-10-18T16:10:26.377 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~76s 2023-10-18T16:10:26.377 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~157s 2023-10-18T16:10:26.377 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.f is failed for ~110s 2023-10-18T16:10:28.568 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:10:28.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.687+0000 7f2ef8025700 1 Processor -- start 2023-10-18T16:10:28.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.687+0000 7f2ef8025700 1 -- start start 2023-10-18T16:10:28.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.688+0000 7f2ef8025700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2ef00a6e80 0x7f2ef0064ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:28.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.688+0000 7f2ef8025700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2ef014d8f0 0x7f2ef014dd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:28.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.688+0000 7f2ef8025700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ef0065390 con 0x7f2ef0151e00 2023-10-18T16:10:28.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.688+0000 7f2ef8025700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ef00a87f0 con 0x7f2ef00a6e80 2023-10-18T16:10:28.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.688+0000 7f2ef8025700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef00a8ad0 con 0x7f2ef014d8f0 2023-10-18T16:10:28.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.688+0000 7f2ef5dc1700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2ef00a6e80 0x7f2ef0064ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:28.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.688+0000 7f2ef5dc1700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2ef00a6e80 0x7f2ef0064ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33774/0 (socket says 172.21.15.88:33774) 2023-10-18T16:10:28.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.688+0000 7f2ef5dc1700 1 -- 172.21.15.88:0/2456299615 learned_addr learned my addr 172.21.15.88:0/2456299615 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:10:28.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.688+0000 7f2ef5dc1700 1 -- 172.21.15.88:0/2456299615 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2ef014d8f0 msgr2=0x7f2ef014dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:10:28.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.688+0000 7f2ef5dc1700 1 --2- 172.21.15.88:0/2456299615 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2ef014d8f0 0x7f2ef014dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:10:28.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.688+0000 7f2ef5dc1700 1 -- 172.21.15.88:0/2456299615 >> v1:172.21.15.151:6789/0 conn(0x7f2ef0151e00 legacy=0x7f2ef0152290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:10:28.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.688+0000 7f2ef5dc1700 1 -- 172.21.15.88:0/2456299615 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ef00a8db0 con 0x7f2ef00a6e80 2023-10-18T16:10:28.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.689+0000 7f2ef5dc1700 1 --2- 172.21.15.88:0/2456299615 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ef00a6e80 0x7f2ef0064ea0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f2eec006fd0 tx=0x7f2eec0090f0).ready entity=mon.0 client_cookie=11789beec4a8b3a9 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:10:28.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.689+0000 7f2ef4dbf700 1 -- 172.21.15.88:0/2456299615 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eec00b600 con 0x7f2ef00a6e80 2023-10-18T16:10:28.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.689+0000 7f2ef4dbf700 1 -- 172.21.15.88:0/2456299615 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2eec00b760 con 0x7f2ef00a6e80 2023-10-18T16:10:28.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.689+0000 7f2ef4dbf700 1 -- 172.21.15.88:0/2456299615 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eec00bb00 con 0x7f2ef00a6e80 2023-10-18T16:10:28.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.689+0000 7f2ef8025700 1 -- 172.21.15.88:0/2456299615 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ef00a6e80 msgr2=0x7f2ef0064ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:28.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.689+0000 7f2ef8025700 1 --2- 172.21.15.88:0/2456299615 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ef00a6e80 0x7f2ef0064ea0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f2eec006fd0 tx=0x7f2eec0090f0).stop 2023-10-18T16:10:28.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.689+0000 7f2ef8025700 1 -- 172.21.15.88:0/2456299615 shutdown_connections 2023-10-18T16:10:28.697 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.689+0000 7f2ef8025700 1 --2- 172.21.15.88:0/2456299615 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ef00a6e80 0x7f2ef0064ea0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:28.697 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.689+0000 7f2ef8025700 1 --2- 172.21.15.88:0/2456299615 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2ef014d8f0 0x7f2ef014dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:10:28.697 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.689+0000 7f2ef8025700 1 -- 172.21.15.88:0/2456299615 >> 172.21.15.88:0/2456299615 conn(0x7f2ef0069d80 msgr2=0x7f2ef006a190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:10:28.697 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.689+0000 7f2ef8025700 1 -- 172.21.15.88:0/2456299615 shutdown_connections 2023-10-18T16:10:28.697 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.689+0000 7f2ef8025700 1 -- 172.21.15.88:0/2456299615 wait complete. 2023-10-18T16:10:28.700 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.690+0000 7f2ef8025700 1 Processor -- start 2023-10-18T16:10:28.700 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.690+0000 7f2ef8025700 1 -- start start 2023-10-18T16:10:28.700 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.691+0000 7f2ef8025700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ef00a6e80 0x7f2ef014a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:28.700 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.691+0000 7f2ef8025700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2ef014d8f0 0x7f2ef014a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:28.700 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.691+0000 7f2ef8025700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2ef0151e00 0x7f2ef0150110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:28.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.691+0000 7f2ef8025700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef0153000 con 0x7f2ef00a6e80 2023-10-18T16:10:28.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.691+0000 7f2ef8025700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef014b0e0 con 0x7f2ef014d8f0 2023-10-18T16:10:28.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.691+0000 7f2ef8025700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef01523f0 con 0x7f2ef0151e00 2023-10-18T16:10:28.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.691+0000 7f2ef5dc1700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ef00a6e80 0x7f2ef014a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:28.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.691+0000 7f2ef5dc1700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ef00a6e80 0x7f2ef014a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33784/0 (socket says 172.21.15.88:33784) 2023-10-18T16:10:28.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.691+0000 7f2ef5dc1700 1 -- 172.21.15.88:0/1429217465 learned_addr learned my addr 172.21.15.88:0/1429217465 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:10:28.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.691+0000 7f2ef65c2700 1 --2- 172.21.15.88:0/1429217465 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2ef0151e00 0x7f2ef0150110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:28.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.692+0000 7f2ef55c0700 1 --2- 172.21.15.88:0/1429217465 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2ef014d8f0 0x7f2ef014a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:28.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.692+0000 7f2ef5dc1700 1 -- 172.21.15.88:0/1429217465 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2ef0151e00 msgr2=0x7f2ef0150110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:28.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.692+0000 7f2ef5dc1700 1 --2- 172.21.15.88:0/1429217465 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2ef0151e00 0x7f2ef0150110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:28.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.692+0000 7f2ef5dc1700 1 -- 172.21.15.88:0/1429217465 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2ef014d8f0 msgr2=0x7f2ef014a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:28.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.692+0000 7f2ef5dc1700 1 --2- 172.21.15.88:0/1429217465 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2ef014d8f0 0x7f2ef014a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:28.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.692+0000 7f2ef5dc1700 1 -- 172.21.15.88:0/1429217465 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eec009370 con 0x7f2ef00a6e80 2023-10-18T16:10:28.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.692+0000 7f2ef55c0700 1 --2- 172.21.15.88:0/1429217465 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2ef014d8f0 0x7f2ef014a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-10-18T16:10:28.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.692+0000 7f2ef65c2700 1 --2- 172.21.15.88:0/1429217465 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2ef0151e00 0x7f2ef0150110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-10-18T16:10:28.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.692+0000 7f2ef5dc1700 1 --2- 172.21.15.88:0/1429217465 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ef00a6e80 0x7f2ef014a1d0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f2eec009170 tx=0x7f2eec017000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:10:28.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.692+0000 7f2ee5ffb700 1 -- 172.21.15.88:0/1429217465 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eec002f70 con 0x7f2ef00a6e80 2023-10-18T16:10:28.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.692+0000 7f2ee5ffb700 1 -- 172.21.15.88:0/1429217465 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2eec00b8d0 con 0x7f2ef00a6e80 2023-10-18T16:10:28.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.692+0000 7f2ef8025700 1 -- 172.21.15.88:0/1429217465 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ef01526d0 con 0x7f2ef00a6e80 2023-10-18T16:10:28.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.692+0000 7f2ee5ffb700 1 -- 172.21.15.88:0/1429217465 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eec024470 con 0x7f2ef00a6e80 2023-10-18T16:10:28.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.692+0000 7f2ef8025700 1 -- 172.21.15.88:0/1429217465 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ef0152c90 con 0x7f2ef00a6e80 2023-10-18T16:10:28.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.693+0000 7f2ef8025700 1 -- 172.21.15.88:0/1429217465 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ef0152e50 con 0x7f2ef00a6e80 2023-10-18T16:10:28.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.693+0000 7f2ee5ffb700 1 -- 172.21.15.88:0/1429217465 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f2eec021070 con 0x7f2ef00a6e80 2023-10-18T16:10:28.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.694+0000 7f2ee5ffb700 1 --2- 172.21.15.88:0/1429217465 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f2ed405f150 0x7f2ed4061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:28.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.694+0000 7f2ee5ffb700 1 -- 172.21.15.88:0/1429217465 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6725+0+0 (secure 0 0 0) 0x7f2eec07c050 con 0x7f2ef00a6e80 2023-10-18T16:10:28.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.694+0000 7f2ef55c0700 1 --2- 172.21.15.88:0/1429217465 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f2ed405f150 0x7f2ed4061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:28.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.694+0000 7f2ef55c0700 1 --2- 172.21.15.88:0/1429217465 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f2ed405f150 0x7f2ed4061610 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f2ef00c79c0 tx=0x7f2ee0008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:10:28.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.698+0000 7f2ee5ffb700 1 -- 172.21.15.88:0/1429217465 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f2eec04d160 con 0x7f2ef00a6e80 2023-10-18T16:10:28.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.897+0000 7f2ef8025700 1 -- 172.21.15.88:0/1429217465 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2ef006c4e0 con 0x7f2ef00a6e80 2023-10-18T16:10:28.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.897+0000 7f2ee5ffb700 1 -- 172.21.15.88:0/1429217465 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7f2eec029070 con 0x7f2ef00a6e80 2023-10-18T16:10:28.902 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:10:28.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.899+0000 7f2ef8025700 1 -- 172.21.15.88:0/1429217465 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f2ed405f150 msgr2=0x7f2ed4061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:28.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.899+0000 7f2ef8025700 1 --2- 172.21.15.88:0/1429217465 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f2ed405f150 0x7f2ed4061610 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f2ef00c79c0 tx=0x7f2ee0008040).stop 2023-10-18T16:10:28.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.899+0000 7f2ef8025700 1 -- 172.21.15.88:0/1429217465 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ef00a6e80 msgr2=0x7f2ef014a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:28.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.899+0000 7f2ef8025700 1 --2- 172.21.15.88:0/1429217465 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ef00a6e80 0x7f2ef014a1d0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f2eec009170 tx=0x7f2eec017000).stop 2023-10-18T16:10:28.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.899+0000 7f2ef8025700 1 -- 172.21.15.88:0/1429217465 shutdown_connections 2023-10-18T16:10:28.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.899+0000 7f2ef8025700 1 --2- 172.21.15.88:0/1429217465 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f2ed405f150 0x7f2ed4061610 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:28.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.899+0000 7f2ef8025700 1 --2- 172.21.15.88:0/1429217465 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2ef0151e00 0x7f2ef0150110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:28.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.899+0000 7f2ef8025700 1 --2- 172.21.15.88:0/1429217465 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ef00a6e80 0x7f2ef014a1d0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:28.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.899+0000 7f2ef8025700 1 --2- 172.21.15.88:0/1429217465 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2ef014d8f0 0x7f2ef014a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:28.905 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.899+0000 7f2ef8025700 1 -- 172.21.15.88:0/1429217465 >> 172.21.15.88:0/1429217465 conn(0x7f2ef0069d80 msgr2=0x7f2ef00c5970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:10:28.905 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.899+0000 7f2ef8025700 1 -- 172.21.15.88:0/1429217465 shutdown_connections 2023-10-18T16:10:28.905 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:28.899+0000 7f2ef8025700 1 -- 172.21.15.88:0/1429217465 wait complete. 2023-10-18T16:10:28.914 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:10:31.915 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:10:32.039 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.036+0000 7f95a327f700 1 Processor -- start 2023-10-18T16:10:32.039 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.036+0000 7f95a327f700 1 -- start start 2023-10-18T16:10:32.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.036+0000 7f95a327f700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f959c0a9050 0x7f959c153ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:32.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.036+0000 7f95a327f700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f959c064150 con 0x7f959c14d8c0 2023-10-18T16:10:32.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.036+0000 7f95a327f700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f959c153a40 con 0x7f959c065360 2023-10-18T16:10:32.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.036+0000 7f95a327f700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f959c063e70 con 0x7f959c0a9050 2023-10-18T16:10:32.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.037+0000 7f95a101b700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f959c14d8c0 0x7f959c064440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:48500/0 (socket says 172.21.15.88:48500) 2023-10-18T16:10:32.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.037+0000 7f95a101b700 1 -- 172.21.15.88:0/4034199193 learned_addr learned my addr 172.21.15.88:0/4034199193 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:10:32.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.037+0000 7f95a081a700 1 --2- 172.21.15.88:0/4034199193 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f959c0a9050 0x7f959c153ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:32.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.037+0000 7f959bfff700 1 -- 172.21.15.88:0/4034199193 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2874369932 0 0) 0x7f959c064150 con 0x7f959c14d8c0 2023-10-18T16:10:32.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.037+0000 7f959bfff700 1 -- 172.21.15.88:0/4034199193 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95840033f0 con 0x7f959c14d8c0 2023-10-18T16:10:32.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.037+0000 7f959bfff700 1 -- 172.21.15.88:0/4034199193 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1440315684 0 0) 0x7f95840033f0 con 0x7f959c14d8c0 2023-10-18T16:10:32.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.037+0000 7f959bfff700 1 -- 172.21.15.88:0/4034199193 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f959c0a9050 msgr2=0x7f959c153ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:10:32.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.037+0000 7f959bfff700 1 --2- 172.21.15.88:0/4034199193 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f959c0a9050 0x7f959c153ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:32.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.037+0000 7f959bfff700 1 -- 172.21.15.88:0/4034199193 >> v1:172.21.15.151:6789/0 conn(0x7f959c065360 legacy=0x7f959c15acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:32.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.037+0000 7f959bfff700 1 -- 172.21.15.88:0/4034199193 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f959c0c72c0 con 0x7f959c14d8c0 2023-10-18T16:10:32.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.037+0000 7f959bfff700 1 -- 172.21.15.88:0/4034199193 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441715075 0 0) 0x7f958c0017f0 con 0x7f959c14d8c0 2023-10-18T16:10:32.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.037+0000 7f959bfff700 1 -- 172.21.15.88:0/4034199193 >> v1:172.21.15.88:6789/0 conn(0x7f959c14d8c0 legacy=0x7f959c064440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:32.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.037+0000 7f959bfff700 1 --2- 172.21.15.88:0/4034199193 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9584003910 0x7f9584003cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:32.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.037+0000 7f959bfff700 1 --2- 172.21.15.88:0/4034199193 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9584004690 0x7f9584004b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:32.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.037+0000 7f959bfff700 1 --2- 172.21.15.88:0/4034199193 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f959c0a9050 0x7f9584005000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:32.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.037+0000 7f959bfff700 1 -- 172.21.15.88:0/4034199193 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f959c063e70 con 0x7f9584004690 2023-10-18T16:10:32.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.037+0000 7f959bfff700 1 -- 172.21.15.88:0/4034199193 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f958400b930 con 0x7f9584003910 2023-10-18T16:10:32.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.037+0000 7f959bfff700 1 -- 172.21.15.88:0/4034199193 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f958400baf0 con 0x7f959c0a9050 2023-10-18T16:10:32.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.037+0000 7f95a081a700 1 --2- 172.21.15.88:0/4034199193 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9584004690 0x7f9584004b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:32.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.037+0000 7f95a101b700 1 --2- 172.21.15.88:0/4034199193 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9584003910 0x7f9584003cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:32.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.037+0000 7f95a181c700 1 --2- 172.21.15.88:0/4034199193 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f959c0a9050 0x7f9584005000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:32.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.038+0000 7f95a081a700 1 -- 172.21.15.88:0/4034199193 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f959c0a9050 msgr2=0x7f9584005000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:10:32.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.038+0000 7f95a081a700 1 --2- 172.21.15.88:0/4034199193 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f959c0a9050 0x7f9584005000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:32.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.038+0000 7f95a081a700 1 -- 172.21.15.88:0/4034199193 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9584003910 msgr2=0x7f9584003cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:10:32.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.038+0000 7f95a081a700 1 --2- 172.21.15.88:0/4034199193 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9584003910 0x7f9584003cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:32.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.038+0000 7f95a081a700 1 -- 172.21.15.88:0/4034199193 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95840033f0 con 0x7f9584004690 2023-10-18T16:10:32.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.038+0000 7f95a081a700 1 --2- 172.21.15.88:0/4034199193 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9584004690 0x7f9584004b10 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f9590009a60 tx=0x7f9590009060).ready entity=mon.0 client_cookie=b903778a412ffd8 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:10:32.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.038+0000 7f959bfff700 1 -- 172.21.15.88:0/4034199193 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9590009ea0 con 0x7f9584004690 2023-10-18T16:10:32.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.038+0000 7f959bfff700 1 -- 172.21.15.88:0/4034199193 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9590009430 con 0x7f9584004690 2023-10-18T16:10:32.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.038+0000 7f959bfff700 1 -- 172.21.15.88:0/4034199193 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f959000e870 con 0x7f9584004690 2023-10-18T16:10:32.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.038+0000 7f95a327f700 1 -- 172.21.15.88:0/4034199193 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9584004690 msgr2=0x7f9584004b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:32.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.038+0000 7f95a327f700 1 --2- 172.21.15.88:0/4034199193 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9584004690 0x7f9584004b10 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f9590009a60 tx=0x7f9590009060).stop 2023-10-18T16:10:32.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.038+0000 7f95a181c700 1 -- 172.21.15.88:0/4034199193 reap_dead start 2023-10-18T16:10:32.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.038+0000 7f95a327f700 1 -- 172.21.15.88:0/4034199193 shutdown_connections 2023-10-18T16:10:32.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.038+0000 7f95a327f700 1 -- 172.21.15.88:0/4034199193 >> 172.21.15.88:0/4034199193 conn(0x7f959c06a810 msgr2=0x7f959c0c6a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:10:32.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.038+0000 7f95a327f700 1 -- 172.21.15.88:0/4034199193 shutdown_connections 2023-10-18T16:10:32.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.038+0000 7f95a327f700 1 -- 172.21.15.88:0/4034199193 wait complete. 2023-10-18T16:10:32.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.040+0000 7f95a327f700 1 Processor -- start 2023-10-18T16:10:32.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.040+0000 7f95a327f700 1 -- start start 2023-10-18T16:10:32.055 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.040+0000 7f95a327f700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9584004690 0x7f959c0aadd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:32.055 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.040+0000 7f95a327f700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f959c0ab300 0x7f959c1637f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:32.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.040+0000 7f95a327f700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f959c163ce0 0x7f959c0a9cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:32.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.040+0000 7f95a327f700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f959c164140 con 0x7f9584004690 2023-10-18T16:10:32.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.040+0000 7f95a327f700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f959c0ac280 con 0x7f959c163ce0 2023-10-18T16:10:32.057 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.040+0000 7f95a327f700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f959c164370 con 0x7f959c0ab300 2023-10-18T16:10:32.057 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.041+0000 7f95a101b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9584004690 0x7f959c0aadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:32.057 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.041+0000 7f95a101b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9584004690 0x7f959c0aadd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33794/0 (socket says 172.21.15.88:33794) 2023-10-18T16:10:32.057 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.041+0000 7f95a101b700 1 -- 172.21.15.88:0/1581558959 learned_addr learned my addr 172.21.15.88:0/1581558959 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:10:32.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.041+0000 7f95a181c700 1 --2- 172.21.15.88:0/1581558959 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f959c163ce0 0x7f959c0a9cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:32.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.041+0000 7f95a081a700 1 --2- 172.21.15.88:0/1581558959 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f959c0ab300 0x7f959c1637f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:32.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.041+0000 7f95a101b700 1 -- 172.21.15.88:0/1581558959 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f959c0ab300 msgr2=0x7f959c1637f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:32.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.041+0000 7f95a101b700 1 --2- 172.21.15.88:0/1581558959 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f959c0ab300 0x7f959c1637f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:32.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.041+0000 7f95a101b700 1 -- 172.21.15.88:0/1581558959 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f959c163ce0 msgr2=0x7f959c0a9cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:32.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.041+0000 7f95a101b700 1 --2- 172.21.15.88:0/1581558959 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f959c163ce0 0x7f959c0a9cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:32.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.041+0000 7f95a101b700 1 -- 172.21.15.88:0/1581558959 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f959c0c0cb0 con 0x7f9584004690 2023-10-18T16:10:32.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.041+0000 7f95a101b700 1 --2- 172.21.15.88:0/1581558959 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9584004690 0x7f959c0aadd0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f958c007710 tx=0x7f958c001d90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:10:32.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.041+0000 7f95997fa700 1 -- 172.21.15.88:0/1581558959 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f958c00f8d0 con 0x7f9584004690 2023-10-18T16:10:32.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.041+0000 7f95997fa700 1 -- 172.21.15.88:0/1581558959 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f958c00fa30 con 0x7f9584004690 2023-10-18T16:10:32.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.041+0000 7f95a327f700 1 -- 172.21.15.88:0/1581558959 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f959c0c0f90 con 0x7f9584004690 2023-10-18T16:10:32.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.041+0000 7f95997fa700 1 -- 172.21.15.88:0/1581558959 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f958c00fdd0 con 0x7f9584004690 2023-10-18T16:10:32.061 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.041+0000 7f95a327f700 1 -- 172.21.15.88:0/1581558959 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f959c0c65b0 con 0x7f9584004690 2023-10-18T16:10:32.061 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.042+0000 7f95a327f700 1 -- 172.21.15.88:0/1581558959 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f959c0ac880 con 0x7f9584004690 2023-10-18T16:10:32.061 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.042+0000 7f95997fa700 1 -- 172.21.15.88:0/1581558959 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f958c012560 con 0x7f9584004690 2023-10-18T16:10:32.061 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.043+0000 7f95997fa700 1 --2- 172.21.15.88:0/1581558959 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f958805f150 0x7f9588061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:32.062 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.043+0000 7f95997fa700 1 -- 172.21.15.88:0/1581558959 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6725+0+0 (secure 0 0 0) 0x7f958c07a110 con 0x7f9584004690 2023-10-18T16:10:32.062 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.043+0000 7f95a081a700 1 --2- 172.21.15.88:0/1581558959 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f958805f150 0x7f9588061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:32.062 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.044+0000 7f95a081a700 1 --2- 172.21.15.88:0/1581558959 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f958805f150 0x7f9588061610 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f959c153330 tx=0x7f959001a000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:10:32.062 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.048+0000 7f95997fa700 1 -- 172.21.15.88:0/1581558959 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f959c0ac880 con 0x7f9584004690 2023-10-18T16:10:32.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.248+0000 7f95a327f700 1 -- 172.21.15.88:0/1581558959 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f959c0ac880 con 0x7f9584004690 2023-10-18T16:10:32.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.248+0000 7f95997fa700 1 -- 172.21.15.88:0/1581558959 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7f959c0ac880 con 0x7f9584004690 2023-10-18T16:10:32.251 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:10:32.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.250+0000 7f95a327f700 1 -- 172.21.15.88:0/1581558959 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f958805f150 msgr2=0x7f9588061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:32.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.250+0000 7f95a327f700 1 --2- 172.21.15.88:0/1581558959 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f958805f150 0x7f9588061610 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f959c153330 tx=0x7f959001a000).stop 2023-10-18T16:10:32.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.250+0000 7f95a327f700 1 -- 172.21.15.88:0/1581558959 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9584004690 msgr2=0x7f959c0aadd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:32.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.250+0000 7f95a327f700 1 --2- 172.21.15.88:0/1581558959 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9584004690 0x7f959c0aadd0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f958c007710 tx=0x7f958c001d90).stop 2023-10-18T16:10:32.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.250+0000 7f95a327f700 1 -- 172.21.15.88:0/1581558959 shutdown_connections 2023-10-18T16:10:32.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.250+0000 7f95a327f700 1 --2- 172.21.15.88:0/1581558959 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f958805f150 0x7f9588061610 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:32.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.250+0000 7f95a327f700 1 --2- 172.21.15.88:0/1581558959 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f959c163ce0 0x7f959c0a9cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:32.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.250+0000 7f95a327f700 1 --2- 172.21.15.88:0/1581558959 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9584004690 0x7f959c0aadd0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:32.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.250+0000 7f95a327f700 1 --2- 172.21.15.88:0/1581558959 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f959c0ab300 0x7f959c1637f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:32.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.250+0000 7f95a327f700 1 -- 172.21.15.88:0/1581558959 >> 172.21.15.88:0/1581558959 conn(0x7f959c06a810 msgr2=0x7f959c14cf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:10:32.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.250+0000 7f95a327f700 1 -- 172.21.15.88:0/1581558959 shutdown_connections 2023-10-18T16:10:32.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:32.250+0000 7f95a327f700 1 -- 172.21.15.88:0/1581558959 wait complete. 2023-10-18T16:10:32.264 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:10:32.686 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~57s 2023-10-18T16:10:32.686 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~70s 2023-10-18T16:10:32.687 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~102s 2023-10-18T16:10:32.687 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~83s 2023-10-18T16:10:32.687 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~164s 2023-10-18T16:10:32.687 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.f is failed for ~116s 2023-10-18T16:10:35.265 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:10:35.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.385+0000 7f1d8f482700 1 Processor -- start 2023-10-18T16:10:35.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.385+0000 7f1d8f482700 1 -- start start 2023-10-18T16:10:35.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.386+0000 7f1d8f482700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1d88068710 0x7f1d88068af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:35.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.386+0000 7f1d8f482700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f1d88064130 0x7f1d88150150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:35.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.386+0000 7f1d8f482700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d88066920 con 0x7f1d880678d0 2023-10-18T16:10:35.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.386+0000 7f1d8f482700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d880674c0 con 0x7f1d88068710 2023-10-18T16:10:35.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.386+0000 7f1d8f482700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d88150720 con 0x7f1d88064130 2023-10-18T16:10:35.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.386+0000 7f1d8e480700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1d88068710 0x7f1d88068af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:35.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.386+0000 7f1d8e480700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1d88068710 0x7f1d88068af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33804/0 (socket says 172.21.15.88:33804) 2023-10-18T16:10:35.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.386+0000 7f1d8e480700 1 -- 172.21.15.88:0/817329541 learned_addr learned my addr 172.21.15.88:0/817329541 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:10:35.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.386+0000 7f1d8e480700 1 -- 172.21.15.88:0/817329541 >> v2:172.21.15.151:3300/0 conn(0x7f1d88064130 msgr2=0x7f1d88150150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:10:35.393 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.386+0000 7f1d8e480700 1 --2- 172.21.15.88:0/817329541 >> v2:172.21.15.151:3300/0 conn(0x7f1d88064130 0x7f1d88150150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:10:35.393 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.386+0000 7f1d8e480700 1 -- 172.21.15.88:0/817329541 >> v1:172.21.15.151:6789/0 conn(0x7f1d880678d0 legacy=0x7f1d88066db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:10:35.393 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.386+0000 7f1d8e480700 1 -- 172.21.15.88:0/817329541 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d88150a00 con 0x7f1d88068710 2023-10-18T16:10:35.393 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.387+0000 7f1d8e480700 1 --2- 172.21.15.88:0/817329541 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d88068710 0x7f1d88068af0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f1d88068150 tx=0x7f1d800090f0).ready entity=mon.0 client_cookie=e2e39db00f805fbe server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:10:35.393 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.387+0000 7f1d8d47e700 1 -- 172.21.15.88:0/817329541 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d8000b630 con 0x7f1d88068710 2023-10-18T16:10:35.394 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.387+0000 7f1d8d47e700 1 -- 172.21.15.88:0/817329541 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d8000b790 con 0x7f1d88068710 2023-10-18T16:10:35.394 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.387+0000 7f1d8d47e700 1 -- 172.21.15.88:0/817329541 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d8000bb10 con 0x7f1d88068710 2023-10-18T16:10:35.394 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.387+0000 7f1d8f482700 1 -- 172.21.15.88:0/817329541 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d88068710 msgr2=0x7f1d88068af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:35.394 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.387+0000 7f1d8f482700 1 --2- 172.21.15.88:0/817329541 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d88068710 0x7f1d88068af0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f1d88068150 tx=0x7f1d800090f0).stop 2023-10-18T16:10:35.394 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.387+0000 7f1d8f482700 1 -- 172.21.15.88:0/817329541 shutdown_connections 2023-10-18T16:10:35.395 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.387+0000 7f1d8f482700 1 --2- 172.21.15.88:0/817329541 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d88068710 0x7f1d88068af0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:35.395 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.387+0000 7f1d8f482700 1 --2- 172.21.15.88:0/817329541 >> v2:172.21.15.151:3300/0 conn(0x7f1d88064130 0x7f1d88150150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:10:35.395 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.387+0000 7f1d8f482700 1 -- 172.21.15.88:0/817329541 >> 172.21.15.88:0/817329541 conn(0x7f1d880c4770 msgr2=0x7f1d880c4b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:10:35.395 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.387+0000 7f1d8f482700 1 -- 172.21.15.88:0/817329541 shutdown_connections 2023-10-18T16:10:35.395 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.387+0000 7f1d8f482700 1 -- 172.21.15.88:0/817329541 wait complete. 2023-10-18T16:10:35.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.388+0000 7f1d8f482700 1 Processor -- start 2023-10-18T16:10:35.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.388+0000 7f1d8f482700 1 -- start start 2023-10-18T16:10:35.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.389+0000 7f1d8f482700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d88064130 0x7f1d88151fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:35.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.389+0000 7f1d8f482700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1d880678d0 0x7f1d881524a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:35.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.389+0000 7f1d8f482700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d88068710 0x7f1d880a9a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:35.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.389+0000 7f1d8f482700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d88151b20 con 0x7f1d88064130 2023-10-18T16:10:35.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.389+0000 7f1d8f482700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d880beb00 con 0x7f1d88068710 2023-10-18T16:10:35.400 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.389+0000 7f1d8f482700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d88152990 con 0x7f1d880678d0 2023-10-18T16:10:35.400 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.389+0000 7f1d8e480700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d88064130 0x7f1d88151fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:35.400 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.389+0000 7f1d8e480700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d88064130 0x7f1d88151fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33810/0 (socket says 172.21.15.88:33810) 2023-10-18T16:10:35.400 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.389+0000 7f1d8e480700 1 -- 172.21.15.88:0/174287767 learned_addr learned my addr 172.21.15.88:0/174287767 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:10:35.400 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.389+0000 7f1d8ec81700 1 --2- 172.21.15.88:0/174287767 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d88068710 0x7f1d880a9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:35.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.389+0000 7f1d8dc7f700 1 --2- 172.21.15.88:0/174287767 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1d880678d0 0x7f1d881524a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:35.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.389+0000 7f1d8e480700 1 -- 172.21.15.88:0/174287767 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1d880678d0 msgr2=0x7f1d881524a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:35.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.389+0000 7f1d8e480700 1 --2- 172.21.15.88:0/174287767 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1d880678d0 0x7f1d881524a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:35.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.389+0000 7f1d8e480700 1 -- 172.21.15.88:0/174287767 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d88068710 msgr2=0x7f1d880a9a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:35.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.389+0000 7f1d8e480700 1 --2- 172.21.15.88:0/174287767 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d88068710 0x7f1d880a9a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:35.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.389+0000 7f1d8e480700 1 -- 172.21.15.88:0/174287767 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d80009370 con 0x7f1d88064130 2023-10-18T16:10:35.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.390+0000 7f1d8e480700 1 --2- 172.21.15.88:0/174287767 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d88064130 0x7f1d88151fb0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f1d80009130 tx=0x7f1d80015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:10:35.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.390+0000 7f1d867fc700 1 -- 172.21.15.88:0/174287767 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d8000b900 con 0x7f1d88064130 2023-10-18T16:10:35.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.390+0000 7f1d867fc700 1 -- 172.21.15.88:0/174287767 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d8000ba60 con 0x7f1d88064130 2023-10-18T16:10:35.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.390+0000 7f1d8f482700 1 -- 172.21.15.88:0/174287767 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d880c2ee0 con 0x7f1d88064130 2023-10-18T16:10:35.403 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.390+0000 7f1d867fc700 1 -- 172.21.15.88:0/174287767 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d8001b660 con 0x7f1d88064130 2023-10-18T16:10:35.403 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.390+0000 7f1d8f482700 1 -- 172.21.15.88:0/174287767 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d88153d00 con 0x7f1d88064130 2023-10-18T16:10:35.403 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.391+0000 7f1d8f482700 1 -- 172.21.15.88:0/174287767 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d88069b50 con 0x7f1d88064130 2023-10-18T16:10:35.403 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.391+0000 7f1d867fc700 1 -- 172.21.15.88:0/174287767 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f1d8001ba10 con 0x7f1d88064130 2023-10-18T16:10:35.403 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.391+0000 7f1d867fc700 1 --2- 172.21.15.88:0/174287767 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f1d7405f120 0x7f1d740615e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:35.404 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.392+0000 7f1d867fc700 1 -- 172.21.15.88:0/174287767 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6725+0+0 (secure 0 0 0) 0x7f1d8007bdd0 con 0x7f1d88064130 2023-10-18T16:10:35.404 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.392+0000 7f1d8dc7f700 1 --2- 172.21.15.88:0/174287767 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f1d7405f120 0x7f1d740615e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:35.404 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.392+0000 7f1d8dc7f700 1 --2- 172.21.15.88:0/174287767 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f1d7405f120 0x7f1d740615e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f1d88149800 tx=0x7f1d7c008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:10:35.404 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.396+0000 7f1d867fc700 1 -- 172.21.15.88:0/174287767 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f1d8004ce40 con 0x7f1d88064130 2023-10-18T16:10:35.599 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.596+0000 7f1d8f482700 1 -- 172.21.15.88:0/174287767 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f1d88150ce0 con 0x7f1d88064130 2023-10-18T16:10:35.600 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.596+0000 7f1d867fc700 1 -- 172.21.15.88:0/174287767 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7f1d8007d020 con 0x7f1d88064130 2023-10-18T16:10:35.600 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:10:35.601 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.598+0000 7f1d8f482700 1 -- 172.21.15.88:0/174287767 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f1d7405f120 msgr2=0x7f1d740615e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:35.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.598+0000 7f1d8f482700 1 --2- 172.21.15.88:0/174287767 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f1d7405f120 0x7f1d740615e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f1d88149800 tx=0x7f1d7c008040).stop 2023-10-18T16:10:35.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.598+0000 7f1d8f482700 1 -- 172.21.15.88:0/174287767 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d88064130 msgr2=0x7f1d88151fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:35.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.598+0000 7f1d8f482700 1 --2- 172.21.15.88:0/174287767 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d88064130 0x7f1d88151fb0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f1d80009130 tx=0x7f1d80015000).stop 2023-10-18T16:10:35.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.598+0000 7f1d8f482700 1 -- 172.21.15.88:0/174287767 shutdown_connections 2023-10-18T16:10:35.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.598+0000 7f1d8f482700 1 --2- 172.21.15.88:0/174287767 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f1d7405f120 0x7f1d740615e0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:35.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.598+0000 7f1d8f482700 1 --2- 172.21.15.88:0/174287767 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d88068710 0x7f1d880a9a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:35.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.598+0000 7f1d8f482700 1 --2- 172.21.15.88:0/174287767 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d88064130 0x7f1d88151fb0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:35.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.598+0000 7f1d8f482700 1 --2- 172.21.15.88:0/174287767 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1d880678d0 0x7f1d881524a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:35.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.598+0000 7f1d8f482700 1 -- 172.21.15.88:0/174287767 >> 172.21.15.88:0/174287767 conn(0x7f1d880c4770 msgr2=0x7f1d88149d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:10:35.604 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.598+0000 7f1d8f482700 1 -- 172.21.15.88:0/174287767 shutdown_connections 2023-10-18T16:10:35.604 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:35.598+0000 7f1d8f482700 1 -- 172.21.15.88:0/174287767 wait complete. 2023-10-18T16:10:35.613 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:10:38.572 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-18T16:10:38.575 DEBUG:teuthology.orchestra.run.smithi151:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-18T16:10:38.614 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:10:38.736 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.733+0000 7f46db59e700 1 Processor -- start 2023-10-18T16:10:38.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.733+0000 7f46db59e700 1 -- start start 2023-10-18T16:10:38.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.734+0000 7f46db59e700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f46dc063d70 0x7f46dc0c6e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:38.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.734+0000 7f46db59e700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f46dc0c2400 0x7f46dc0c2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:38.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.734+0000 7f46db59e700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46dc151020 con 0x7f46dc0c6a60 2023-10-18T16:10:38.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.734+0000 7f46db59e700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46dc150090 con 0x7f46dc0c2400 2023-10-18T16:10:38.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.734+0000 7f46db59e700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46dc0c2d80 con 0x7f46dc063d70 2023-10-18T16:10:38.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.734+0000 7f46da59c700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f46dc0c6a60 0x7f46dc153660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:40758/0 (socket says 172.21.15.88:40758) 2023-10-18T16:10:38.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.734+0000 7f46da59c700 1 -- 172.21.15.88:0/3303214837 learned_addr learned my addr 172.21.15.88:0/3303214837 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:10:38.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.734+0000 7f46dad9d700 1 --2- 172.21.15.88:0/3303214837 >> v2:172.21.15.151:3300/0 conn(0x7f46dc0c2400 0x7f46dc0c2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:38.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.734+0000 7f46d9d9b700 1 --2- 172.21.15.88:0/3303214837 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f46dc063d70 0x7f46dc0c6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:38.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.734+0000 7f46d959a700 1 -- 172.21.15.88:0/3303214837 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1189835579 0 0) 0x7f46cc001110 con 0x7f46dc0c6a60 2023-10-18T16:10:38.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.734+0000 7f46d959a700 1 -- 172.21.15.88:0/3303214837 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46c40033f0 con 0x7f46dc0c6a60 2023-10-18T16:10:38.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.734+0000 7f46d959a700 1 -- 172.21.15.88:0/3303214837 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2993535989 0 0) 0x7f46c40033f0 con 0x7f46dc0c6a60 2023-10-18T16:10:38.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.735+0000 7f46d959a700 1 -- 172.21.15.88:0/3303214837 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f46dc063d70 msgr2=0x7f46dc0c6e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:10:38.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.735+0000 7f46d959a700 1 --2- 172.21.15.88:0/3303214837 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f46dc063d70 0x7f46dc0c6e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:38.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.735+0000 7f46d959a700 1 -- 172.21.15.88:0/3303214837 >> v2:172.21.15.151:3300/0 conn(0x7f46dc0c2400 msgr2=0x7f46dc0c2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:10:38.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.735+0000 7f46d959a700 1 --2- 172.21.15.88:0/3303214837 >> v2:172.21.15.151:3300/0 conn(0x7f46dc0c2400 0x7f46dc0c2890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:38.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.735+0000 7f46d959a700 1 -- 172.21.15.88:0/3303214837 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46dc150b90 con 0x7f46dc0c6a60 2023-10-18T16:10:38.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.735+0000 7f46d959a700 1 -- 172.21.15.88:0/3303214837 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441715075 0 0) 0x7f46cc001600 con 0x7f46dc0c6a60 2023-10-18T16:10:38.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.735+0000 7f46d959a700 1 -- 172.21.15.88:0/3303214837 >> v1:172.21.15.88:6789/0 conn(0x7f46dc0c6a60 legacy=0x7f46dc153660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:38.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.735+0000 7f46d959a700 1 --2- 172.21.15.88:0/3303214837 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f46c4003910 0x7f46c4003d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:38.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.735+0000 7f46d959a700 1 --2- 172.21.15.88:0/3303214837 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f46dc063d70 0x7f46c4004690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:38.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.735+0000 7f46d959a700 1 --2- 172.21.15.88:0/3303214837 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f46c4004b80 0x7f46c4005000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:38.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.735+0000 7f46d959a700 1 -- 172.21.15.88:0/3303214837 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c40044c0 con 0x7f46c4004b80 2023-10-18T16:10:38.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.735+0000 7f46d959a700 1 -- 172.21.15.88:0/3303214837 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c400b8e0 con 0x7f46c4003910 2023-10-18T16:10:38.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.735+0000 7f46d959a700 1 -- 172.21.15.88:0/3303214837 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46c400baa0 con 0x7f46dc063d70 2023-10-18T16:10:38.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.735+0000 7f46dad9d700 1 --2- 172.21.15.88:0/3303214837 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f46c4004b80 0x7f46c4005000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:38.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.735+0000 7f46d9d9b700 1 --2- 172.21.15.88:0/3303214837 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f46dc063d70 0x7f46c4004690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:38.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.735+0000 7f46da59c700 1 --2- 172.21.15.88:0/3303214837 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f46c4003910 0x7f46c4003d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:38.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.735+0000 7f46dad9d700 1 -- 172.21.15.88:0/3303214837 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f46dc063d70 msgr2=0x7f46c4004690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:10:38.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.735+0000 7f46dad9d700 1 --2- 172.21.15.88:0/3303214837 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f46dc063d70 0x7f46c4004690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:38.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.735+0000 7f46dad9d700 1 -- 172.21.15.88:0/3303214837 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f46c4003910 msgr2=0x7f46c4003d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:10:38.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.735+0000 7f46dad9d700 1 --2- 172.21.15.88:0/3303214837 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f46c4003910 0x7f46c4003d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:38.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.735+0000 7f46dad9d700 1 -- 172.21.15.88:0/3303214837 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46c40033f0 con 0x7f46c4004b80 2023-10-18T16:10:38.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.735+0000 7f46dad9d700 1 --2- 172.21.15.88:0/3303214837 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f46c4004b80 0x7f46c4005000 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f46d400e210 tx=0x7f46d400b040).ready entity=mon.0 client_cookie=234ba6b76a2e7d66 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:10:38.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.735+0000 7f46d959a700 1 -- 172.21.15.88:0/3303214837 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d4009ac0 con 0x7f46c4004b80 2023-10-18T16:10:38.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.735+0000 7f46d959a700 1 -- 172.21.15.88:0/3303214837 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46d400f040 con 0x7f46c4004b80 2023-10-18T16:10:38.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.735+0000 7f46d959a700 1 -- 172.21.15.88:0/3303214837 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d400ecd0 con 0x7f46c4004b80 2023-10-18T16:10:38.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.736+0000 7f46db59e700 1 -- 172.21.15.88:0/3303214837 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f46c4004b80 msgr2=0x7f46c4005000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:38.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.736+0000 7f46db59e700 1 --2- 172.21.15.88:0/3303214837 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f46c4004b80 0x7f46c4005000 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f46d400e210 tx=0x7f46d400b040).stop 2023-10-18T16:10:38.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.736+0000 7f46dad9d700 1 -- 172.21.15.88:0/3303214837 reap_dead start 2023-10-18T16:10:38.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.736+0000 7f46db59e700 1 -- 172.21.15.88:0/3303214837 shutdown_connections 2023-10-18T16:10:38.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.736+0000 7f46db59e700 1 -- 172.21.15.88:0/3303214837 >> 172.21.15.88:0/3303214837 conn(0x7f46dc05e050 msgr2=0x7f46dc05e460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:10:38.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.736+0000 7f46db59e700 1 -- 172.21.15.88:0/3303214837 shutdown_connections 2023-10-18T16:10:38.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.736+0000 7f46db59e700 1 -- 172.21.15.88:0/3303214837 wait complete. 2023-10-18T16:10:38.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.737+0000 7f46db59e700 1 Processor -- start 2023-10-18T16:10:38.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.737+0000 7f46db59e700 1 -- start start 2023-10-18T16:10:38.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.738+0000 7f46db59e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f46c4004b80 0x7f46dc149ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:38.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.738+0000 7f46db59e700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f46dc1499a0 0x7f46dc0ba810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:38.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.738+0000 7f46db59e700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f46dc1606e0 0x7f46dc0bde20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:38.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.738+0000 7f46db59e700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46dc160b60 con 0x7f46dc1606e0 2023-10-18T16:10:38.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.738+0000 7f46db59e700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46dc0bd680 con 0x7f46c4004b80 2023-10-18T16:10:38.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.738+0000 7f46db59e700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46dc14fd30 con 0x7f46dc1499a0 2023-10-18T16:10:38.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.738+0000 7f46dad9d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f46dc1606e0 0x7f46dc0bde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:38.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.738+0000 7f46dad9d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f46dc1606e0 0x7f46dc0bde20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46264/0 (socket says 172.21.15.88:46264) 2023-10-18T16:10:38.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.738+0000 7f46dad9d700 1 -- 172.21.15.88:0/3462440837 learned_addr learned my addr 172.21.15.88:0/3462440837 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:10:38.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.738+0000 7f46d9d9b700 1 --2- 172.21.15.88:0/3462440837 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f46dc1499a0 0x7f46dc0ba810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:38.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.738+0000 7f46da59c700 1 --2- 172.21.15.88:0/3462440837 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f46c4004b80 0x7f46dc149ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:38.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.738+0000 7f46dad9d700 1 -- 172.21.15.88:0/3462440837 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f46dc1499a0 msgr2=0x7f46dc0ba810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:38.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.738+0000 7f46dad9d700 1 --2- 172.21.15.88:0/3462440837 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f46dc1499a0 0x7f46dc0ba810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:38.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.738+0000 7f46dad9d700 1 -- 172.21.15.88:0/3462440837 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f46c4004b80 msgr2=0x7f46dc149ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:38.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.738+0000 7f46dad9d700 1 --2- 172.21.15.88:0/3462440837 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f46c4004b80 0x7f46dc149ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:38.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.738+0000 7f46dad9d700 1 -- 172.21.15.88:0/3462440837 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46dc14fff0 con 0x7f46dc1606e0 2023-10-18T16:10:38.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.738+0000 7f46dad9d700 1 --2- 172.21.15.88:0/3462440837 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f46dc1606e0 0x7f46dc0bde20 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7f46d4014000 tx=0x7f46d401a000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:10:38.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.738+0000 7f46ca7fc700 1 -- 172.21.15.88:0/3462440837 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d4021030 con 0x7f46dc1606e0 2023-10-18T16:10:38.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.738+0000 7f46ca7fc700 1 -- 172.21.15.88:0/3462440837 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46d4009ac0 con 0x7f46dc1606e0 2023-10-18T16:10:38.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.738+0000 7f46db59e700 1 -- 172.21.15.88:0/3462440837 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46dc154870 con 0x7f46dc1606e0 2023-10-18T16:10:38.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.738+0000 7f46ca7fc700 1 -- 172.21.15.88:0/3462440837 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d4013dc0 con 0x7f46dc1606e0 2023-10-18T16:10:38.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.738+0000 7f46db59e700 1 -- 172.21.15.88:0/3462440837 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46dc0bd2f0 con 0x7f46dc1606e0 2023-10-18T16:10:38.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.739+0000 7f46db59e700 1 -- 172.21.15.88:0/3462440837 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46dc0bd830 con 0x7f46dc1606e0 2023-10-18T16:10:38.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.740+0000 7f46ca7fc700 1 -- 172.21.15.88:0/3462440837 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f46d4024070 con 0x7f46dc1606e0 2023-10-18T16:10:38.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.741+0000 7f46ca7fc700 1 --2- 172.21.15.88:0/3462440837 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f46bc05f150 0x7f46bc061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:38.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.741+0000 7f46ca7fc700 1 -- 172.21.15.88:0/3462440837 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6725+0+0 (secure 0 0 0) 0x7f46d4022070 con 0x7f46dc1606e0 2023-10-18T16:10:38.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.741+0000 7f46da59c700 1 --2- 172.21.15.88:0/3462440837 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f46bc05f150 0x7f46bc061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:38.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.741+0000 7f46da59c700 1 --2- 172.21.15.88:0/3462440837 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f46bc05f150 0x7f46bc061610 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f46cc007a00 tx=0x7f46dc150b00).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:10:38.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.744+0000 7f46ca7fc700 1 -- 172.21.15.88:0/3462440837 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f46d404d3d0 con 0x7f46dc1606e0 2023-10-18T16:10:38.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.941+0000 7f46db59e700 1 -- 172.21.15.88:0/3462440837 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f46dc0bd830 con 0x7f46dc1606e0 2023-10-18T16:10:38.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.941+0000 7f46ca7fc700 1 -- 172.21.15.88:0/3462440837 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7f46dc0bd830 con 0x7f46dc1606e0 2023-10-18T16:10:38.945 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:10:38.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.943+0000 7f46db59e700 1 -- 172.21.15.88:0/3462440837 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f46bc05f150 msgr2=0x7f46bc061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:38.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.943+0000 7f46db59e700 1 --2- 172.21.15.88:0/3462440837 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f46bc05f150 0x7f46bc061610 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f46cc007a00 tx=0x7f46dc150b00).stop 2023-10-18T16:10:38.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.943+0000 7f46db59e700 1 -- 172.21.15.88:0/3462440837 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f46dc1606e0 msgr2=0x7f46dc0bde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:38.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.943+0000 7f46db59e700 1 --2- 172.21.15.88:0/3462440837 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f46dc1606e0 0x7f46dc0bde20 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7f46d4014000 tx=0x7f46d401a000).stop 2023-10-18T16:10:38.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.943+0000 7f46db59e700 1 -- 172.21.15.88:0/3462440837 shutdown_connections 2023-10-18T16:10:38.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.943+0000 7f46db59e700 1 --2- 172.21.15.88:0/3462440837 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f46bc05f150 0x7f46bc061610 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:38.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.943+0000 7f46db59e700 1 --2- 172.21.15.88:0/3462440837 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f46dc1606e0 0x7f46dc0bde20 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:38.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.943+0000 7f46db59e700 1 --2- 172.21.15.88:0/3462440837 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f46c4004b80 0x7f46dc149ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:38.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.943+0000 7f46db59e700 1 --2- 172.21.15.88:0/3462440837 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f46dc1499a0 0x7f46dc0ba810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:38.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.943+0000 7f46db59e700 1 -- 172.21.15.88:0/3462440837 >> 172.21.15.88:0/3462440837 conn(0x7f46dc05e050 msgr2=0x7f46dc14bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:10:38.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.943+0000 7f46db59e700 1 -- 172.21.15.88:0/3462440837 shutdown_connections 2023-10-18T16:10:38.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:38.943+0000 7f46db59e700 1 -- 172.21.15.88:0/3462440837 wait complete. 2023-10-18T16:10:38.958 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:10:38.997 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~63s 2023-10-18T16:10:38.998 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~76s 2023-10-18T16:10:38.998 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~109s 2023-10-18T16:10:38.999 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~89s 2023-10-18T16:10:38.999 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~170s 2023-10-18T16:10:38.999 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.f is failed for ~122s 2023-10-18T16:10:41.959 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:10:42.079 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.076+0000 7fe7048e3700 1 Processor -- start 2023-10-18T16:10:42.080 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.076+0000 7fe7048e3700 1 -- start start 2023-10-18T16:10:42.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.077+0000 7fe7048e3700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fe700153010 0x7fe7001533f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:42.084 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.077+0000 7fe7048e3700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe700064d70 0x7fe700151ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:42.084 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.077+0000 7fe7048e3700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7000a9670 con 0x7fe7001523d0 2023-10-18T16:10:42.084 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.077+0000 7fe7048e3700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe70014df50 con 0x7fe700153010 2023-10-18T16:10:42.084 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.077+0000 7fe7048e3700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe700151a50 con 0x7fe700064d70 2023-10-18T16:10:42.084 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.077+0000 7fe6fed9d700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fe7001523d0 0x7fe7000c36c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:40760/0 (socket says 172.21.15.88:40760) 2023-10-18T16:10:42.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.077+0000 7fe6fed9d700 1 -- 172.21.15.88:0/1422462713 learned_addr learned my addr 172.21.15.88:0/1422462713 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:10:42.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.077+0000 7fe6fe59c700 1 --2- 172.21.15.88:0/1422462713 >> v2:172.21.15.151:3300/0 conn(0x7fe700153010 0x7fe7001533f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:42.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.077+0000 7fe6fd59a700 1 -- 172.21.15.88:0/1422462713 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1709575644 0 0) 0x7fe7000a9670 con 0x7fe7001523d0 2023-10-18T16:10:42.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.077+0000 7fe6fd59a700 1 -- 172.21.15.88:0/1422462713 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6e80033f0 con 0x7fe7001523d0 2023-10-18T16:10:42.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.077+0000 7fe6fdd9b700 1 --2- 172.21.15.88:0/1422462713 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe700064d70 0x7fe700151ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:42.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.077+0000 7fe6fd59a700 1 -- 172.21.15.88:0/1422462713 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 785029134 0 0) 0x7fe6e80033f0 con 0x7fe7001523d0 2023-10-18T16:10:42.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.077+0000 7fe6fd59a700 1 -- 172.21.15.88:0/1422462713 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe700064d70 msgr2=0x7fe700151ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:10:42.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.077+0000 7fe6fd59a700 1 --2- 172.21.15.88:0/1422462713 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe700064d70 0x7fe700151ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:42.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.077+0000 7fe6fd59a700 1 -- 172.21.15.88:0/1422462713 >> v2:172.21.15.151:3300/0 conn(0x7fe700153010 msgr2=0x7fe7001533f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:10:42.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.078+0000 7fe6fd59a700 1 --2- 172.21.15.88:0/1422462713 >> v2:172.21.15.151:3300/0 conn(0x7fe700153010 0x7fe7001533f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:42.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.078+0000 7fe6fd59a700 1 -- 172.21.15.88:0/1422462713 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7000a8af0 con 0x7fe7001523d0 2023-10-18T16:10:42.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.078+0000 7fe6fd59a700 1 -- 172.21.15.88:0/1422462713 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441715075 0 0) 0x7fe6f80017f0 con 0x7fe7001523d0 2023-10-18T16:10:42.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.078+0000 7fe6fd59a700 1 -- 172.21.15.88:0/1422462713 >> v1:172.21.15.88:6789/0 conn(0x7fe7001523d0 legacy=0x7fe7000c36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:42.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.078+0000 7fe6fd59a700 1 --2- 172.21.15.88:0/1422462713 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe6e80038e0 0x7fe6e8003d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:42.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.078+0000 7fe6fd59a700 1 --2- 172.21.15.88:0/1422462713 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe6e8004690 0x7fe6e8004b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:42.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.078+0000 7fe6fd59a700 1 --2- 172.21.15.88:0/1422462713 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe700064d70 0x7fe6e8005000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:42.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.078+0000 7fe6fd59a700 1 -- 172.21.15.88:0/1422462713 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe70014df50 con 0x7fe6e8004690 2023-10-18T16:10:42.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.078+0000 7fe6fd59a700 1 -- 172.21.15.88:0/1422462713 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe700151a50 con 0x7fe6e80038e0 2023-10-18T16:10:42.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.078+0000 7fe6fd59a700 1 -- 172.21.15.88:0/1422462713 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e800b8e0 con 0x7fe700064d70 2023-10-18T16:10:42.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.078+0000 7fe6fdd9b700 1 --2- 172.21.15.88:0/1422462713 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe6e8004690 0x7fe6e8004b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:42.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.078+0000 7fe6fe59c700 1 --2- 172.21.15.88:0/1422462713 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe6e80038e0 0x7fe6e8003d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:42.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.078+0000 7fe6fed9d700 1 --2- 172.21.15.88:0/1422462713 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe700064d70 0x7fe6e8005000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:42.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.078+0000 7fe6fdd9b700 1 -- 172.21.15.88:0/1422462713 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe700064d70 msgr2=0x7fe6e8005000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:10:42.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.078+0000 7fe6fdd9b700 1 --2- 172.21.15.88:0/1422462713 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe700064d70 0x7fe6e8005000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:42.090 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.078+0000 7fe6fdd9b700 1 -- 172.21.15.88:0/1422462713 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe6e80038e0 msgr2=0x7fe6e8003d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:10:42.090 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.078+0000 7fe6fdd9b700 1 --2- 172.21.15.88:0/1422462713 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe6e80038e0 0x7fe6e8003d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:42.090 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.078+0000 7fe6fdd9b700 1 -- 172.21.15.88:0/1422462713 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6e80033f0 con 0x7fe6e8004690 2023-10-18T16:10:42.090 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.078+0000 7fe6fdd9b700 1 --2- 172.21.15.88:0/1422462713 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe6e8004690 0x7fe6e8004b10 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7fe6f4009a80 tx=0x7fe6f4009060).ready entity=mon.0 client_cookie=ecd4650c251e1e88 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:10:42.091 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.078+0000 7fe6fd59a700 1 -- 172.21.15.88:0/1422462713 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6f4009430 con 0x7fe6e8004690 2023-10-18T16:10:42.091 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.078+0000 7fe6fd59a700 1 -- 172.21.15.88:0/1422462713 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6f400e640 con 0x7fe6e8004690 2023-10-18T16:10:42.091 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.078+0000 7fe6fd59a700 1 -- 172.21.15.88:0/1422462713 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6f400e9c0 con 0x7fe6e8004690 2023-10-18T16:10:42.091 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.079+0000 7fe7048e3700 1 -- 172.21.15.88:0/1422462713 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe6e8004690 msgr2=0x7fe6e8004b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:42.091 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.079+0000 7fe7048e3700 1 --2- 172.21.15.88:0/1422462713 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe6e8004690 0x7fe6e8004b10 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7fe6f4009a80 tx=0x7fe6f4009060).stop 2023-10-18T16:10:42.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.079+0000 7fe6fed9d700 1 -- 172.21.15.88:0/1422462713 reap_dead start 2023-10-18T16:10:42.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.079+0000 7fe7048e3700 1 -- 172.21.15.88:0/1422462713 shutdown_connections 2023-10-18T16:10:42.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.079+0000 7fe7048e3700 1 -- 172.21.15.88:0/1422462713 >> 172.21.15.88:0/1422462713 conn(0x7fe700069d80 msgr2=0x7fe70006a190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:10:42.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.079+0000 7fe7048e3700 1 -- 172.21.15.88:0/1422462713 shutdown_connections 2023-10-18T16:10:42.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.079+0000 7fe7048e3700 1 -- 172.21.15.88:0/1422462713 wait complete. 2023-10-18T16:10:42.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.080+0000 7fe7048e3700 1 Processor -- start 2023-10-18T16:10:42.093 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.080+0000 7fe7048e3700 1 -- start start 2023-10-18T16:10:42.095 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.081+0000 7fe7048e3700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe6e8004690 0x7fe7000c3c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:42.096 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.081+0000 7fe7048e3700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7000a2480 0x7fe7000bd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:42.096 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.081+0000 7fe7048e3700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe7000bd610 0x7fe70014ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:42.096 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.081+0000 7fe7048e3700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe700163340 con 0x7fe7000a2480 2023-10-18T16:10:42.096 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.081+0000 7fe7048e3700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe700164160 con 0x7fe7000bd610 2023-10-18T16:10:42.096 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.081+0000 7fe7048e3700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe700163550 con 0x7fe6e8004690 2023-10-18T16:10:42.096 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.081+0000 7fe6fdd9b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7000a2480 0x7fe7000bd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:42.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.081+0000 7fe6fdd9b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7000a2480 0x7fe7000bd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46292/0 (socket says 172.21.15.88:46292) 2023-10-18T16:10:42.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.081+0000 7fe6fdd9b700 1 -- 172.21.15.88:0/2852991334 learned_addr learned my addr 172.21.15.88:0/2852991334 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:10:42.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.081+0000 7fe6fed9d700 1 --2- 172.21.15.88:0/2852991334 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe7000bd610 0x7fe70014ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:42.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.081+0000 7fe6fe59c700 1 --2- 172.21.15.88:0/2852991334 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe6e8004690 0x7fe7000c3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:42.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.081+0000 7fe6fdd9b700 1 -- 172.21.15.88:0/2852991334 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe6e8004690 msgr2=0x7fe7000c3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:42.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.081+0000 7fe6fdd9b700 1 --2- 172.21.15.88:0/2852991334 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe6e8004690 0x7fe7000c3c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:42.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.081+0000 7fe6fdd9b700 1 -- 172.21.15.88:0/2852991334 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe7000bd610 msgr2=0x7fe70014ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:42.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.081+0000 7fe6fdd9b700 1 --2- 172.21.15.88:0/2852991334 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe7000bd610 0x7fe70014ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:42.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.081+0000 7fe6fdd9b700 1 -- 172.21.15.88:0/2852991334 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe700163830 con 0x7fe7000a2480 2023-10-18T16:10:42.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.081+0000 7fe6fdd9b700 1 --2- 172.21.15.88:0/2852991334 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7000a2480 0x7fe7000bd120 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7fe6e80055c0 tx=0x7fe6f401a000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:10:42.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.081+0000 7fe6ee7fc700 1 -- 172.21.15.88:0/2852991334 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6f400ed30 con 0x7fe7000a2480 2023-10-18T16:10:42.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.081+0000 7fe6ee7fc700 1 -- 172.21.15.88:0/2852991334 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6f400ee90 con 0x7fe7000a2480 2023-10-18T16:10:42.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.081+0000 7fe7048e3700 1 -- 172.21.15.88:0/2852991334 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe700163b10 con 0x7fe7000a2480 2023-10-18T16:10:42.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.081+0000 7fe6ee7fc700 1 -- 172.21.15.88:0/2852991334 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6f4026440 con 0x7fe7000a2480 2023-10-18T16:10:42.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.081+0000 7fe7048e3700 1 -- 172.21.15.88:0/2852991334 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7000a9f80 con 0x7fe7000a2480 2023-10-18T16:10:42.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.083+0000 7fe7048e3700 1 -- 172.21.15.88:0/2852991334 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7000c4560 con 0x7fe7000a2480 2023-10-18T16:10:42.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.083+0000 7fe6ee7fc700 1 -- 172.21.15.88:0/2852991334 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7fe6f402f440 con 0x7fe7000a2480 2023-10-18T16:10:42.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.083+0000 7fe6ee7fc700 1 --2- 172.21.15.88:0/2852991334 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fe6e005f150 0x7fe6e0061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:42.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.083+0000 7fe6ee7fc700 1 -- 172.21.15.88:0/2852991334 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6725+0+0 (secure 0 0 0) 0x7fe6f407c0d0 con 0x7fe7000a2480 2023-10-18T16:10:42.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.083+0000 7fe6fe59c700 1 --2- 172.21.15.88:0/2852991334 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fe6e005f150 0x7fe6e0061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:42.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.084+0000 7fe6fe59c700 1 --2- 172.21.15.88:0/2852991334 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fe6e005f150 0x7fe6e0061610 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7fe6f0006d80 tx=0x7fe6f0008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:10:42.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.087+0000 7fe6ee7fc700 1 -- 172.21.15.88:0/2852991334 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7fe6f404d0e0 con 0x7fe7000a2480 2023-10-18T16:10:42.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.287+0000 7fe7048e3700 1 -- 172.21.15.88:0/2852991334 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe700163fb0 con 0x7fe7000a2480 2023-10-18T16:10:42.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.288+0000 7fe6ee7fc700 1 -- 172.21.15.88:0/2852991334 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7fe6f4016080 con 0x7fe7000a2480 2023-10-18T16:10:42.293 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:10:42.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.290+0000 7fe7048e3700 1 -- 172.21.15.88:0/2852991334 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fe6e005f150 msgr2=0x7fe6e0061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:42.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.290+0000 7fe7048e3700 1 --2- 172.21.15.88:0/2852991334 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fe6e005f150 0x7fe6e0061610 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7fe6f0006d80 tx=0x7fe6f0008040).stop 2023-10-18T16:10:42.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.290+0000 7fe7048e3700 1 -- 172.21.15.88:0/2852991334 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7000a2480 msgr2=0x7fe7000bd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:42.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.290+0000 7fe7048e3700 1 --2- 172.21.15.88:0/2852991334 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7000a2480 0x7fe7000bd120 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7fe6e80055c0 tx=0x7fe6f401a000).stop 2023-10-18T16:10:42.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.290+0000 7fe7048e3700 1 -- 172.21.15.88:0/2852991334 shutdown_connections 2023-10-18T16:10:42.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.290+0000 7fe7048e3700 1 --2- 172.21.15.88:0/2852991334 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fe6e005f150 0x7fe6e0061610 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:42.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.290+0000 7fe7048e3700 1 --2- 172.21.15.88:0/2852991334 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe7000bd610 0x7fe70014ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:42.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.290+0000 7fe7048e3700 1 --2- 172.21.15.88:0/2852991334 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe6e8004690 0x7fe7000c3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:42.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.290+0000 7fe7048e3700 1 --2- 172.21.15.88:0/2852991334 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7000a2480 0x7fe7000bd120 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:42.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.290+0000 7fe7048e3700 1 -- 172.21.15.88:0/2852991334 >> 172.21.15.88:0/2852991334 conn(0x7fe700069d80 msgr2=0x7fe7000ac240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:10:42.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.290+0000 7fe7048e3700 1 -- 172.21.15.88:0/2852991334 shutdown_connections 2023-10-18T16:10:42.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:42.290+0000 7fe7048e3700 1 -- 172.21.15.88:0/2852991334 wait complete. 2023-10-18T16:10:42.305 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:10:45.307 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:10:45.309 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~69s 2023-10-18T16:10:45.310 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~82s 2023-10-18T16:10:45.310 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~115s 2023-10-18T16:10:45.310 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~95s 2023-10-18T16:10:45.310 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~176s 2023-10-18T16:10:45.310 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.f is failed for ~129s 2023-10-18T16:10:45.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.424+0000 7fdccffff700 1 Processor -- start 2023-10-18T16:10:45.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.424+0000 7fdccffff700 1 -- start start 2023-10-18T16:10:45.430 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.425+0000 7fdccffff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fdcd00be1d0 0x7fdcd00be5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:45.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.425+0000 7fdccffff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdcd006a9e0 0x7fdcd0063820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:45.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.425+0000 7fdccffff700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fdcd00c0310 0x7fdcd0063d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:45.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.425+0000 7fdccffff700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcd014d420 con 0x7fdcd00be1d0 2023-10-18T16:10:45.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.425+0000 7fdccffff700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcd00bd960 con 0x7fdcd00c0310 2023-10-18T16:10:45.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.425+0000 7fdccffff700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd014c810 con 0x7fdcd006a9e0 2023-10-18T16:10:45.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.425+0000 7fdcceffd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fdcd00be1d0 0x7fdcd00be5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:45.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.425+0000 7fdcceffd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fdcd00be1d0 0x7fdcd00be5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46300/0 (socket says 172.21.15.88:46300) 2023-10-18T16:10:45.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.425+0000 7fdcceffd700 1 -- 172.21.15.88:0/453986532 learned_addr learned my addr 172.21.15.88:0/453986532 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:10:45.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.425+0000 7fdcceffd700 1 -- 172.21.15.88:0/453986532 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdcd006a9e0 msgr2=0x7fdcd0063820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:10:45.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.425+0000 7fdcce7fc700 1 --2- 172.21.15.88:0/453986532 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdcd006a9e0 0x7fdcd0063820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:45.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.425+0000 7fdccf7fe700 1 --2- 172.21.15.88:0/453986532 >> v2:172.21.15.151:3300/0 conn(0x7fdcd00c0310 0x7fdcd0063d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:45.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.425+0000 7fdcceffd700 1 --2- 172.21.15.88:0/453986532 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdcd006a9e0 0x7fdcd0063820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:45.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.425+0000 7fdcceffd700 1 -- 172.21.15.88:0/453986532 >> v2:172.21.15.151:3300/0 conn(0x7fdcd00c0310 msgr2=0x7fdcd0063d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:10:45.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.425+0000 7fdcceffd700 1 --2- 172.21.15.88:0/453986532 >> v2:172.21.15.151:3300/0 conn(0x7fdcd00c0310 0x7fdcd0063d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:45.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.425+0000 7fdcceffd700 1 -- 172.21.15.88:0/453986532 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcd014caf0 con 0x7fdcd00be1d0 2023-10-18T16:10:45.434 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.425+0000 7fdcceffd700 1 --2- 172.21.15.88:0/453986532 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdcd00be1d0 0x7fdcd00be5b0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7fdcd0066390 tx=0x7fdcc000a950).ready entity=mon.0 client_cookie=b9b623bfe2f4612f server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:10:45.434 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.425+0000 7fdccdffb700 1 -- 172.21.15.88:0/453986532 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc000f070 con 0x7fdcd00be1d0 2023-10-18T16:10:45.434 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.426+0000 7fdccdffb700 1 -- 172.21.15.88:0/453986532 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcc000ce50 con 0x7fdcd00be1d0 2023-10-18T16:10:45.434 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.426+0000 7fdccdffb700 1 -- 172.21.15.88:0/453986532 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc000a2e0 con 0x7fdcd00be1d0 2023-10-18T16:10:45.434 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.426+0000 7fdccffff700 1 -- 172.21.15.88:0/453986532 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdcd00be1d0 msgr2=0x7fdcd00be5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:45.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.426+0000 7fdccffff700 1 --2- 172.21.15.88:0/453986532 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdcd00be1d0 0x7fdcd00be5b0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7fdcd0066390 tx=0x7fdcc000a950).stop 2023-10-18T16:10:45.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.426+0000 7fdccffff700 1 -- 172.21.15.88:0/453986532 shutdown_connections 2023-10-18T16:10:45.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.426+0000 7fdccffff700 1 --2- 172.21.15.88:0/453986532 >> v2:172.21.15.151:3300/0 conn(0x7fdcd00c0310 0x7fdcd0063d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:45.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.426+0000 7fdccffff700 1 --2- 172.21.15.88:0/453986532 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdcd00be1d0 0x7fdcd00be5b0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:45.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.426+0000 7fdccffff700 1 --2- 172.21.15.88:0/453986532 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdcd006a9e0 0x7fdcd0063820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:45.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.426+0000 7fdccffff700 1 -- 172.21.15.88:0/453986532 >> 172.21.15.88:0/453986532 conn(0x7fdcd00c3750 msgr2=0x7fdcd00bf720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:10:45.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.426+0000 7fdccffff700 1 -- 172.21.15.88:0/453986532 shutdown_connections 2023-10-18T16:10:45.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.426+0000 7fdccffff700 1 -- 172.21.15.88:0/453986532 wait complete. 2023-10-18T16:10:45.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.427+0000 7fdccffff700 1 Processor -- start 2023-10-18T16:10:45.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.427+0000 7fdccffff700 1 -- start start 2023-10-18T16:10:45.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.428+0000 7fdccffff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdcd006a9e0 0x7fdcd0146030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:45.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.428+0000 7fdccffff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdcd00be1d0 0x7fdcd0141bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:45.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.428+0000 7fdccffff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdcd00c0310 0x7fdcd01420c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:45.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.428+0000 7fdccffff700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd0145ba0 con 0x7fdcd00be1d0 2023-10-18T16:10:45.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.428+0000 7fdccffff700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd0144a40 con 0x7fdcd006a9e0 2023-10-18T16:10:45.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.428+0000 7fdccffff700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd0144cb0 con 0x7fdcd00c0310 2023-10-18T16:10:45.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.428+0000 7fdcce7fc700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdcd00be1d0 0x7fdcd0141bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:45.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.428+0000 7fdcce7fc700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdcd00be1d0 0x7fdcd0141bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46312/0 (socket says 172.21.15.88:46312) 2023-10-18T16:10:45.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.428+0000 7fdcce7fc700 1 -- 172.21.15.88:0/3610062236 learned_addr learned my addr 172.21.15.88:0/3610062236 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:10:45.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.428+0000 7fdcceffd700 1 --2- 172.21.15.88:0/3610062236 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdcd006a9e0 0x7fdcd0146030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:45.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.428+0000 7fdcce7fc700 1 -- 172.21.15.88:0/3610062236 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdcd00c0310 msgr2=0x7fdcd01420c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:45.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.428+0000 7fdcce7fc700 1 --2- 172.21.15.88:0/3610062236 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdcd00c0310 0x7fdcd01420c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-10-18T16:10:45.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.428+0000 7fdcce7fc700 1 -- 172.21.15.88:0/3610062236 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdcd006a9e0 msgr2=0x7fdcd0146030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:45.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.428+0000 7fdcce7fc700 1 --2- 172.21.15.88:0/3610062236 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdcd006a9e0 0x7fdcd0146030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:45.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.428+0000 7fdcce7fc700 1 -- 172.21.15.88:0/3610062236 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcc000abf0 con 0x7fdcd00be1d0 2023-10-18T16:10:45.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.428+0000 7fdcceffd700 1 --2- 172.21.15.88:0/3610062236 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdcd006a9e0 0x7fdcd0146030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-10-18T16:10:45.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.429+0000 7fdcce7fc700 1 --2- 172.21.15.88:0/3610062236 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdcd00be1d0 0x7fdcd0141bd0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7fdcb800c8b0 tx=0x7fdcb8008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:10:45.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.429+0000 7fdcc6ffd700 1 -- 172.21.15.88:0/3610062236 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcb8010580 con 0x7fdcd00be1d0 2023-10-18T16:10:45.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.429+0000 7fdcc6ffd700 1 -- 172.21.15.88:0/3610062236 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcb80106e0 con 0x7fdcd00be1d0 2023-10-18T16:10:45.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.429+0000 7fdccffff700 1 -- 172.21.15.88:0/3610062236 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcd0144f90 con 0x7fdcd00be1d0 2023-10-18T16:10:45.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.429+0000 7fdcc6ffd700 1 -- 172.21.15.88:0/3610062236 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcb8010a60 con 0x7fdcd00be1d0 2023-10-18T16:10:45.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.429+0000 7fdccffff700 1 -- 172.21.15.88:0/3610062236 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcd0145550 con 0x7fdcd00be1d0 2023-10-18T16:10:45.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.430+0000 7fdccffff700 1 -- 172.21.15.88:0/3610062236 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcd0145d50 con 0x7fdcd00be1d0 2023-10-18T16:10:45.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.430+0000 7fdcc6ffd700 1 -- 172.21.15.88:0/3610062236 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7fdcb801b460 con 0x7fdcd00be1d0 2023-10-18T16:10:45.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.430+0000 7fdcc6ffd700 1 --2- 172.21.15.88:0/3610062236 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fdcb005f150 0x7fdcb0061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:45.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.431+0000 7fdcc6ffd700 1 -- 172.21.15.88:0/3610062236 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6725+0+0 (secure 0 0 0) 0x7fdcb807a420 con 0x7fdcd00be1d0 2023-10-18T16:10:45.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.431+0000 7fdcceffd700 1 --2- 172.21.15.88:0/3610062236 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fdcb005f150 0x7fdcb0061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:45.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.431+0000 7fdcceffd700 1 --2- 172.21.15.88:0/3610062236 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fdcb005f150 0x7fdcb0061610 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fdcd0066390 tx=0x7fdcc000a950).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:10:45.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.435+0000 7fdcc6ffd700 1 -- 172.21.15.88:0/3610062236 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7fdcd0145d50 con 0x7fdcd00be1d0 2023-10-18T16:10:45.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.635+0000 7fdccffff700 1 -- 172.21.15.88:0/3610062236 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fdcd0145d50 con 0x7fdcd00be1d0 2023-10-18T16:10:45.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.636+0000 7fdcc6ffd700 1 -- 172.21.15.88:0/3610062236 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7fdcd0145d50 con 0x7fdcd00be1d0 2023-10-18T16:10:45.640 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:10:45.641 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.637+0000 7fdccffff700 1 -- 172.21.15.88:0/3610062236 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fdcb005f150 msgr2=0x7fdcb0061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:45.642 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.637+0000 7fdccffff700 1 --2- 172.21.15.88:0/3610062236 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fdcb005f150 0x7fdcb0061610 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fdcd0066390 tx=0x7fdcc000a950).stop 2023-10-18T16:10:45.642 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.637+0000 7fdccffff700 1 -- 172.21.15.88:0/3610062236 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdcd00be1d0 msgr2=0x7fdcd0141bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:45.642 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.637+0000 7fdccffff700 1 --2- 172.21.15.88:0/3610062236 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdcd00be1d0 0x7fdcd0141bd0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7fdcb800c8b0 tx=0x7fdcb8008040).stop 2023-10-18T16:10:45.642 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.638+0000 7fdccffff700 1 -- 172.21.15.88:0/3610062236 shutdown_connections 2023-10-18T16:10:45.642 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.638+0000 7fdccffff700 1 --2- 172.21.15.88:0/3610062236 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fdcb005f150 0x7fdcb0061610 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:45.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.638+0000 7fdccffff700 1 --2- 172.21.15.88:0/3610062236 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdcd00c0310 0x7fdcd01420c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-10-18T16:10:45.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.638+0000 7fdccffff700 1 --2- 172.21.15.88:0/3610062236 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdcd006a9e0 0x7fdcd0146030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:45.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.638+0000 7fdccffff700 1 --2- 172.21.15.88:0/3610062236 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdcd00be1d0 0x7fdcd0141bd0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:45.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.638+0000 7fdccffff700 1 -- 172.21.15.88:0/3610062236 >> 172.21.15.88:0/3610062236 conn(0x7fdcd00c3750 msgr2=0x7fdcd00c0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:10:45.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.638+0000 7fdccffff700 1 -- 172.21.15.88:0/3610062236 shutdown_connections 2023-10-18T16:10:45.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:45.638+0000 7fdccffff700 1 -- 172.21.15.88:0/3610062236 wait complete. 2023-10-18T16:10:45.653 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:10:48.654 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:10:48.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.776+0000 7f14ba00f700 1 Processor -- start 2023-10-18T16:10:48.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.776+0000 7f14ba00f700 1 -- start start 2023-10-18T16:10:48.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.777+0000 7f14ba00f700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f14b414d8c0 0x7f14b4064440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:48.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.777+0000 7f14ba00f700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f14b4065360 0x7f14b4064930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:48.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.777+0000 7f14ba00f700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14b4064150 con 0x7f14b40a9050 2023-10-18T16:10:48.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.777+0000 7f14ba00f700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14b4153a40 con 0x7f14b4065360 2023-10-18T16:10:48.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.777+0000 7f14ba00f700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14b4063e70 con 0x7f14b414d8c0 2023-10-18T16:10:48.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.777+0000 7f14b2ffd700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f14b40a9050 0x7f14b4153ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:48578/0 (socket says 172.21.15.88:48578) 2023-10-18T16:10:48.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.777+0000 7f14b2ffd700 1 -- 172.21.15.88:0/1542898243 learned_addr learned my addr 172.21.15.88:0/1542898243 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:10:48.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.777+0000 7f14b37fe700 1 --2- 172.21.15.88:0/1542898243 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f14b414d8c0 0x7f14b4064440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:48.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.777+0000 7f14b3fff700 1 --2- 172.21.15.88:0/1542898243 >> v2:172.21.15.151:3300/0 conn(0x7f14b4065360 0x7f14b4064930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:48.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.777+0000 7f14b27fc700 1 -- 172.21.15.88:0/1542898243 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3462503189 0 0) 0x7f14b4064150 con 0x7f14b40a9050 2023-10-18T16:10:48.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.777+0000 7f14b27fc700 1 -- 172.21.15.88:0/1542898243 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f149c0033f0 con 0x7f14b40a9050 2023-10-18T16:10:48.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.777+0000 7f14b27fc700 1 -- 172.21.15.88:0/1542898243 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1416442102 0 0) 0x7f149c0033f0 con 0x7f14b40a9050 2023-10-18T16:10:48.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.777+0000 7f14b27fc700 1 -- 172.21.15.88:0/1542898243 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f14b414d8c0 msgr2=0x7f14b4064440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:10:48.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.777+0000 7f14b27fc700 1 --2- 172.21.15.88:0/1542898243 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f14b414d8c0 0x7f14b4064440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:48.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.777+0000 7f14b27fc700 1 -- 172.21.15.88:0/1542898243 >> v2:172.21.15.151:3300/0 conn(0x7f14b4065360 msgr2=0x7f14b4064930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:10:48.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.777+0000 7f14b27fc700 1 --2- 172.21.15.88:0/1542898243 >> v2:172.21.15.151:3300/0 conn(0x7f14b4065360 0x7f14b4064930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:48.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.777+0000 7f14b27fc700 1 -- 172.21.15.88:0/1542898243 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14b40c72c0 con 0x7f14b40a9050 2023-10-18T16:10:48.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.777+0000 7f14b27fc700 1 -- 172.21.15.88:0/1542898243 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441715075 0 0) 0x7f14a8001470 con 0x7f14b40a9050 2023-10-18T16:10:48.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.777+0000 7f14b27fc700 1 -- 172.21.15.88:0/1542898243 >> v1:172.21.15.88:6789/0 conn(0x7f14b40a9050 legacy=0x7f14b4153ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:48.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.777+0000 7f14b27fc700 1 --2- 172.21.15.88:0/1542898243 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f149c0038e0 0x7f149c003d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:48.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.777+0000 7f14b27fc700 1 --2- 172.21.15.88:0/1542898243 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f149c004690 0x7f149c004b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:48.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.777+0000 7f14b27fc700 1 --2- 172.21.15.88:0/1542898243 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f149c005000 0x7f149c00b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:48.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.777+0000 7f14b27fc700 1 -- 172.21.15.88:0/1542898243 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14b4153a40 con 0x7f149c0038e0 2023-10-18T16:10:48.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.777+0000 7f14b27fc700 1 -- 172.21.15.88:0/1542898243 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14b4063e70 con 0x7f149c005000 2023-10-18T16:10:48.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.777+0000 7f14b27fc700 1 -- 172.21.15.88:0/1542898243 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f149c00be00 con 0x7f149c004690 2023-10-18T16:10:48.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.777+0000 7f14b3fff700 1 --2- 172.21.15.88:0/1542898243 >> v2:172.21.15.151:3300/0 conn(0x7f14b4065360 0x7f14b4064930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2023-10-18T16:10:48.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.777+0000 7f14b37fe700 1 --2- 172.21.15.88:0/1542898243 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f14b414d8c0 0x7f14b4064440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2023-10-18T16:10:48.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.778+0000 7f14b2ffd700 1 --2- 172.21.15.88:0/1542898243 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f149c004690 0x7f149c004b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:48.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.778+0000 7f14b37fe700 1 --2- 172.21.15.88:0/1542898243 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f149c0038e0 0x7f149c003d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:48.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.778+0000 7f14b3fff700 1 --2- 172.21.15.88:0/1542898243 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f149c005000 0x7f149c00b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:48.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.778+0000 7f14b37fe700 1 -- 172.21.15.88:0/1542898243 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f149c004690 msgr2=0x7f149c004b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:10:48.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.778+0000 7f14b37fe700 1 --2- 172.21.15.88:0/1542898243 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f149c004690 0x7f149c004b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:48.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.778+0000 7f14b37fe700 1 -- 172.21.15.88:0/1542898243 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f149c005000 msgr2=0x7f149c00b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:10:48.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.778+0000 7f14b37fe700 1 --2- 172.21.15.88:0/1542898243 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f149c005000 0x7f149c00b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:48.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.778+0000 7f14b37fe700 1 -- 172.21.15.88:0/1542898243 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f149c0033f0 con 0x7f149c0038e0 2023-10-18T16:10:48.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.778+0000 7f14b37fe700 1 --2- 172.21.15.88:0/1542898243 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f149c0038e0 0x7f149c003d00 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f14b40bdad0 tx=0x7f14a4009060).ready entity=mon.0 client_cookie=efed5e3e51db01cf server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:10:48.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.778+0000 7f14b27fc700 1 -- 172.21.15.88:0/1542898243 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a40094d0 con 0x7f149c0038e0 2023-10-18T16:10:48.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.778+0000 7f14b27fc700 1 -- 172.21.15.88:0/1542898243 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14a400e5e0 con 0x7f149c0038e0 2023-10-18T16:10:48.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.778+0000 7f14b27fc700 1 -- 172.21.15.88:0/1542898243 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a400e9a0 con 0x7f149c0038e0 2023-10-18T16:10:48.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.778+0000 7f14ba00f700 1 -- 172.21.15.88:0/1542898243 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f149c0038e0 msgr2=0x7f149c003d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:48.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.778+0000 7f14ba00f700 1 --2- 172.21.15.88:0/1542898243 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f149c0038e0 0x7f149c003d00 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f14b40bdad0 tx=0x7f14a4009060).stop 2023-10-18T16:10:48.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.779+0000 7f14b3fff700 1 -- 172.21.15.88:0/1542898243 reap_dead start 2023-10-18T16:10:48.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.779+0000 7f14ba00f700 1 -- 172.21.15.88:0/1542898243 shutdown_connections 2023-10-18T16:10:48.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.779+0000 7f14ba00f700 1 -- 172.21.15.88:0/1542898243 >> 172.21.15.88:0/1542898243 conn(0x7f14b406a810 msgr2=0x7f14b40c6a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:10:48.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.779+0000 7f14ba00f700 1 -- 172.21.15.88:0/1542898243 shutdown_connections 2023-10-18T16:10:48.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.779+0000 7f14ba00f700 1 -- 172.21.15.88:0/1542898243 wait complete. 2023-10-18T16:10:48.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.780+0000 7f14ba00f700 1 Processor -- start 2023-10-18T16:10:48.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.780+0000 7f14ba00f700 1 -- start start 2023-10-18T16:10:48.794 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.781+0000 7f14ba00f700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f149c0038e0 0x7f14b40bc090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:48.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.781+0000 7f14ba00f700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f14b40ba890 0x7f14b40bbaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:48.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.781+0000 7f14ba00f700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f14b4160b30 0x7f14b40ac130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:48.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.781+0000 7f14ba00f700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14b414fb90 con 0x7f149c0038e0 2023-10-18T16:10:48.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.781+0000 7f14ba00f700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14b4160160 con 0x7f14b40ba890 2023-10-18T16:10:48.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.781+0000 7f14ba00f700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14b40ba410 con 0x7f14b4160b30 2023-10-18T16:10:48.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.781+0000 7f14b37fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f149c0038e0 0x7f14b40bc090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:48.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.781+0000 7f14b37fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f149c0038e0 0x7f14b40bc090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41230/0 (socket says 172.21.15.88:41230) 2023-10-18T16:10:48.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.781+0000 7f14b37fe700 1 -- 172.21.15.88:0/3612251842 learned_addr learned my addr 172.21.15.88:0/3612251842 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:10:48.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.781+0000 7f14b2ffd700 1 --2- 172.21.15.88:0/3612251842 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f14b40ba890 0x7f14b40bbaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:48.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.781+0000 7f14b3fff700 1 --2- 172.21.15.88:0/3612251842 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f14b4160b30 0x7f14b40ac130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:48.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.781+0000 7f14b37fe700 1 -- 172.21.15.88:0/3612251842 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f14b4160b30 msgr2=0x7f14b40ac130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:48.797 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.781+0000 7f14b37fe700 1 --2- 172.21.15.88:0/3612251842 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f14b4160b30 0x7f14b40ac130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:48.797 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.781+0000 7f14b37fe700 1 -- 172.21.15.88:0/3612251842 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f14b40ba890 msgr2=0x7f14b40bbaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:48.797 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.781+0000 7f14b37fe700 1 --2- 172.21.15.88:0/3612251842 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f14b40ba890 0x7f14b40bbaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:48.797 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.781+0000 7f14b37fe700 1 -- 172.21.15.88:0/3612251842 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14b41606a0 con 0x7f149c0038e0 2023-10-18T16:10:48.797 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.781+0000 7f14b37fe700 1 --2- 172.21.15.88:0/3612251842 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f149c0038e0 0x7f14b40bc090 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f14a4012e10 tx=0x7f14a4019000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:10:48.797 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.781+0000 7f149b7fe700 1 -- 172.21.15.88:0/3612251842 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a4009430 con 0x7f149c0038e0 2023-10-18T16:10:48.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.781+0000 7f149b7fe700 1 -- 172.21.15.88:0/3612251842 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14a4022070 con 0x7f149c0038e0 2023-10-18T16:10:48.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.781+0000 7f14ba00f700 1 -- 172.21.15.88:0/3612251842 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14b4160350 con 0x7f149c0038e0 2023-10-18T16:10:48.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.781+0000 7f149b7fe700 1 -- 172.21.15.88:0/3612251842 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a400ed10 con 0x7f149c0038e0 2023-10-18T16:10:48.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.781+0000 7f14ba00f700 1 -- 172.21.15.88:0/3612251842 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14b40bcf20 con 0x7f149c0038e0 2023-10-18T16:10:48.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.782+0000 7f14ba00f700 1 -- 172.21.15.88:0/3612251842 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14b4160980 con 0x7f149c0038e0 2023-10-18T16:10:48.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.783+0000 7f149b7fe700 1 -- 172.21.15.88:0/3612251842 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f14a401e630 con 0x7f149c0038e0 2023-10-18T16:10:48.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.783+0000 7f149b7fe700 1 --2- 172.21.15.88:0/3612251842 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f14a005f150 0x7f14a0061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:48.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.783+0000 7f149b7fe700 1 -- 172.21.15.88:0/3612251842 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6725+0+0 (secure 0 0 0) 0x7f14a407c130 con 0x7f149c0038e0 2023-10-18T16:10:48.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.783+0000 7f14b2ffd700 1 --2- 172.21.15.88:0/3612251842 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f14a005f150 0x7f14a0061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:48.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.784+0000 7f14b2ffd700 1 --2- 172.21.15.88:0/3612251842 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f14a005f150 0x7f14a0061610 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f14a8000e50 tx=0x7f14a8001e90).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:10:48.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.787+0000 7f149b7fe700 1 -- 172.21.15.88:0/3612251842 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f14a404d1a0 con 0x7f149c0038e0 2023-10-18T16:10:48.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.989+0000 7f14ba00f700 1 -- 172.21.15.88:0/3612251842 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f14b406c4e0 con 0x7f149c0038e0 2023-10-18T16:10:48.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.989+0000 7f149b7fe700 1 -- 172.21.15.88:0/3612251842 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7f14a402c020 con 0x7f149c0038e0 2023-10-18T16:10:48.994 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:10:48.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.991+0000 7f14ba00f700 1 -- 172.21.15.88:0/3612251842 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f14a005f150 msgr2=0x7f14a0061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:48.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.991+0000 7f14ba00f700 1 --2- 172.21.15.88:0/3612251842 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f14a005f150 0x7f14a0061610 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f14a8000e50 tx=0x7f14a8001e90).stop 2023-10-18T16:10:48.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.991+0000 7f14ba00f700 1 -- 172.21.15.88:0/3612251842 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f149c0038e0 msgr2=0x7f14b40bc090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:48.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.991+0000 7f14ba00f700 1 --2- 172.21.15.88:0/3612251842 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f149c0038e0 0x7f14b40bc090 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f14a4012e10 tx=0x7f14a4019000).stop 2023-10-18T16:10:48.997 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.991+0000 7f14ba00f700 1 -- 172.21.15.88:0/3612251842 shutdown_connections 2023-10-18T16:10:48.997 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.992+0000 7f14ba00f700 1 --2- 172.21.15.88:0/3612251842 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f14a005f150 0x7f14a0061610 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:48.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.992+0000 7f14ba00f700 1 --2- 172.21.15.88:0/3612251842 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f14b4160b30 0x7f14b40ac130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:48.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.992+0000 7f14ba00f700 1 --2- 172.21.15.88:0/3612251842 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f149c0038e0 0x7f14b40bc090 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:48.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.992+0000 7f14ba00f700 1 --2- 172.21.15.88:0/3612251842 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f14b40ba890 0x7f14b40bbaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:48.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.992+0000 7f14ba00f700 1 -- 172.21.15.88:0/3612251842 >> 172.21.15.88:0/3612251842 conn(0x7f14b406a810 msgr2=0x7f14b414d010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:10:48.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.992+0000 7f14ba00f700 1 -- 172.21.15.88:0/3612251842 shutdown_connections 2023-10-18T16:10:48.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:48.992+0000 7f14ba00f700 1 -- 172.21.15.88:0/3612251842 wait complete. 2023-10-18T16:10:49.007 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:10:51.620 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~76s 2023-10-18T16:10:51.621 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~89s 2023-10-18T16:10:51.621 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~121s 2023-10-18T16:10:51.621 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~102s 2023-10-18T16:10:51.621 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~183s 2023-10-18T16:10:51.622 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.f is failed for ~135s 2023-10-18T16:10:52.008 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:10:52.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.128+0000 7f162c49d700 1 Processor -- start 2023-10-18T16:10:52.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.128+0000 7f162c49d700 1 -- start start 2023-10-18T16:10:52.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.129+0000 7f162c49d700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f1624153a90 0x7f1624069de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:52.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.129+0000 7f162c49d700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f16240c3b80 0x7f16240c4010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:52.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.129+0000 7f162c49d700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1624153fd0 con 0x7f16240bedb0 2023-10-18T16:10:52.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.129+0000 7f162c49d700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1624151560 con 0x7f1624153a90 2023-10-18T16:10:52.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.129+0000 7f162c49d700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f162414e070 con 0x7f16240c3b80 2023-10-18T16:10:52.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.130+0000 7f1629a38700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f1624153a90 0x7f1624069de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:52.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.130+0000 7f162aa3a700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f16240c3b80 0x7f16240c4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:52.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.130+0000 7f162a239700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f16240bedb0 0x7f1624150e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.88:49558/0 (socket says 172.21.15.88:49558) 2023-10-18T16:10:52.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.130+0000 7f162a239700 1 -- 172.21.15.88:0/1297837842 learned_addr learned my addr 172.21.15.88:0/1297837842 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:10:52.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.130+0000 7f1629237700 1 -- 172.21.15.88:0/1297837842 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2679765600 0 0) 0x7f1624153fd0 con 0x7f16240bedb0 2023-10-18T16:10:52.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.130+0000 7f1629237700 1 -- 172.21.15.88:0/1297837842 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f160c0033f0 con 0x7f16240bedb0 2023-10-18T16:10:52.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.130+0000 7f162aa3a700 1 -- 172.21.15.88:0/1297837842 >> v2:172.21.15.151:3300/0 conn(0x7f1624153a90 msgr2=0x7f1624069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:10:52.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.130+0000 7f162aa3a700 1 --2- 172.21.15.88:0/1297837842 >> v2:172.21.15.151:3300/0 conn(0x7f1624153a90 0x7f1624069de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:52.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.130+0000 7f162aa3a700 1 -- 172.21.15.88:0/1297837842 >> v1:172.21.15.151:6789/0 conn(0x7f16240bedb0 legacy=0x7f1624150e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:52.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.130+0000 7f162aa3a700 1 -- 172.21.15.88:0/1297837842 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16241542b0 con 0x7f16240c3b80 2023-10-18T16:10:52.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.130+0000 7f1629a38700 1 --2- 172.21.15.88:0/1297837842 >> v2:172.21.15.151:3300/0 conn(0x7f1624153a90 0x7f1624069de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2023-10-18T16:10:52.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.131+0000 7f162aa3a700 1 --2- 172.21.15.88:0/1297837842 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f16240c3b80 0x7f16240c4010 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f161c009760 tx=0x7f161c009060).ready entity=mon.2 client_cookie=5381fc9ba7d7f3b4 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:10:52.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.131+0000 7f1629237700 1 -- 172.21.15.88:0/1297837842 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f161c00f510 con 0x7f16240c3b80 2023-10-18T16:10:52.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.131+0000 7f1629237700 1 -- 172.21.15.88:0/1297837842 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f161c00f670 con 0x7f16240c3b80 2023-10-18T16:10:52.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.131+0000 7f1629237700 1 -- 172.21.15.88:0/1297837842 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f161c00f9f0 con 0x7f16240c3b80 2023-10-18T16:10:52.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.131+0000 7f162c49d700 1 -- 172.21.15.88:0/1297837842 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f16240c3b80 msgr2=0x7f16240c4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:52.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.131+0000 7f162c49d700 1 --2- 172.21.15.88:0/1297837842 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f16240c3b80 0x7f16240c4010 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f161c009760 tx=0x7f161c009060).stop 2023-10-18T16:10:52.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.131+0000 7f162c49d700 1 -- 172.21.15.88:0/1297837842 shutdown_connections 2023-10-18T16:10:52.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.131+0000 7f162c49d700 1 --2- 172.21.15.88:0/1297837842 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f16240c3b80 0x7f16240c4010 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:52.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.131+0000 7f162c49d700 1 --2- 172.21.15.88:0/1297837842 >> v2:172.21.15.151:3300/0 conn(0x7f1624153a90 0x7f1624069de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:52.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.131+0000 7f162c49d700 1 -- 172.21.15.88:0/1297837842 >> 172.21.15.88:0/1297837842 conn(0x7f1624066220 msgr2=0x7f162405e050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:10:52.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.131+0000 7f162c49d700 1 -- 172.21.15.88:0/1297837842 shutdown_connections 2023-10-18T16:10:52.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.131+0000 7f162c49d700 1 -- 172.21.15.88:0/1297837842 wait complete. 2023-10-18T16:10:52.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.132+0000 7f162c49d700 1 Processor -- start 2023-10-18T16:10:52.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.132+0000 7f162c49d700 1 -- start start 2023-10-18T16:10:52.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.133+0000 7f162c49d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f16240bedb0 0x7f162414f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:52.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.133+0000 7f162c49d700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f16240c3b80 0x7f16241516b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:52.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.133+0000 7f162c49d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1624153a90 0x7f1624164040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:52.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.133+0000 7f162c49d700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1624161370 con 0x7f1624153a90 2023-10-18T16:10:52.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.133+0000 7f162c49d700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1624152020 con 0x7f16240bedb0 2023-10-18T16:10:52.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.133+0000 7f162c49d700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1624163bb0 con 0x7f16240c3b80 2023-10-18T16:10:52.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.133+0000 7f162aa3a700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1624153a90 0x7f1624164040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:52.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.133+0000 7f162aa3a700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1624153a90 0x7f1624164040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41234/0 (socket says 172.21.15.88:41234) 2023-10-18T16:10:52.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.133+0000 7f162aa3a700 1 -- 172.21.15.88:0/340470937 learned_addr learned my addr 172.21.15.88:0/340470937 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:10:52.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.133+0000 7f1629a38700 1 --2- 172.21.15.88:0/340470937 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f16240c3b80 0x7f16241516b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:52.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.133+0000 7f162a239700 1 --2- 172.21.15.88:0/340470937 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f16240bedb0 0x7f162414f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:52.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.134+0000 7f162aa3a700 1 -- 172.21.15.88:0/340470937 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f16240c3b80 msgr2=0x7f16241516b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:52.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.134+0000 7f162aa3a700 1 --2- 172.21.15.88:0/340470937 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f16240c3b80 0x7f16241516b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:52.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.134+0000 7f162aa3a700 1 -- 172.21.15.88:0/340470937 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f16240bedb0 msgr2=0x7f162414f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:52.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.134+0000 7f162aa3a700 1 --2- 172.21.15.88:0/340470937 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f16240bedb0 0x7f162414f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:52.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.134+0000 7f162aa3a700 1 -- 172.21.15.88:0/340470937 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16180092a0 con 0x7f1624153a90 2023-10-18T16:10:52.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.134+0000 7f162aa3a700 1 --2- 172.21.15.88:0/340470937 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1624153a90 0x7f1624164040 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f161c009d50 tx=0x7f161c017000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:10:52.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.134+0000 7f16227fc700 1 -- 172.21.15.88:0/340470937 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f161c00f7e0 con 0x7f1624153a90 2023-10-18T16:10:52.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.134+0000 7f16227fc700 1 -- 172.21.15.88:0/340470937 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f161c00f940 con 0x7f1624153a90 2023-10-18T16:10:52.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.134+0000 7f162c49d700 1 -- 172.21.15.88:0/340470937 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f161c0092a0 con 0x7f1624153a90 2023-10-18T16:10:52.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.134+0000 7f16227fc700 1 -- 172.21.15.88:0/340470937 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f161c01b640 con 0x7f1624153a90 2023-10-18T16:10:52.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.134+0000 7f162c49d700 1 -- 172.21.15.88:0/340470937 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1624160ad0 con 0x7f1624153a90 2023-10-18T16:10:52.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.135+0000 7f162c49d700 1 -- 172.21.15.88:0/340470937 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f162406c4e0 con 0x7f1624153a90 2023-10-18T16:10:52.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.135+0000 7f16227fc700 1 -- 172.21.15.88:0/340470937 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f161c01b9b0 con 0x7f1624153a90 2023-10-18T16:10:52.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.135+0000 7f16227fc700 1 --2- 172.21.15.88:0/340470937 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f161005f150 0x7f1610061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:52.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.135+0000 7f16227fc700 1 -- 172.21.15.88:0/340470937 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6725+0+0 (secure 0 0 0) 0x7f161c07bf20 con 0x7f1624153a90 2023-10-18T16:10:52.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.136+0000 7f162a239700 1 --2- 172.21.15.88:0/340470937 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f161005f150 0x7f1610061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:52.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.136+0000 7f162a239700 1 --2- 172.21.15.88:0/340470937 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f161005f150 0x7f1610061610 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f1614005ad0 tx=0x7f1614009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:10:52.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.140+0000 7f16227fc700 1 -- 172.21.15.88:0/340470937 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f161c04d040 con 0x7f1624153a90 2023-10-18T16:10:52.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.338+0000 7f162c49d700 1 -- 172.21.15.88:0/340470937 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f1624163cf0 con 0x7f1624153a90 2023-10-18T16:10:52.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.338+0000 7f16227fc700 1 -- 172.21.15.88:0/340470937 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7f161c050660 con 0x7f1624153a90 2023-10-18T16:10:52.342 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:10:52.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.340+0000 7f162c49d700 1 -- 172.21.15.88:0/340470937 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f161005f150 msgr2=0x7f1610061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:52.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.340+0000 7f162c49d700 1 --2- 172.21.15.88:0/340470937 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f161005f150 0x7f1610061610 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f1614005ad0 tx=0x7f1614009040).stop 2023-10-18T16:10:52.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.340+0000 7f162c49d700 1 -- 172.21.15.88:0/340470937 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1624153a90 msgr2=0x7f1624164040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:52.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.340+0000 7f162c49d700 1 --2- 172.21.15.88:0/340470937 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1624153a90 0x7f1624164040 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f161c009d50 tx=0x7f161c017000).stop 2023-10-18T16:10:52.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.340+0000 7f162c49d700 1 -- 172.21.15.88:0/340470937 shutdown_connections 2023-10-18T16:10:52.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.340+0000 7f162c49d700 1 --2- 172.21.15.88:0/340470937 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f161005f150 0x7f1610061610 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:52.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.340+0000 7f162c49d700 1 --2- 172.21.15.88:0/340470937 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1624153a90 0x7f1624164040 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:52.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.340+0000 7f162c49d700 1 --2- 172.21.15.88:0/340470937 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f16240bedb0 0x7f162414f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:52.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.340+0000 7f162c49d700 1 --2- 172.21.15.88:0/340470937 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f16240c3b80 0x7f16241516b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:52.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.340+0000 7f162c49d700 1 -- 172.21.15.88:0/340470937 >> 172.21.15.88:0/340470937 conn(0x7f1624066220 msgr2=0x7f1624069c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:10:52.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.340+0000 7f162c49d700 1 -- 172.21.15.88:0/340470937 shutdown_connections 2023-10-18T16:10:52.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:52.340+0000 7f162c49d700 1 -- 172.21.15.88:0/340470937 wait complete. 2023-10-18T16:10:52.355 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:10:55.356 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:10:55.484 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.481+0000 7f09f6166700 1 Processor -- start 2023-10-18T16:10:55.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.481+0000 7f09f6166700 1 -- start start 2023-10-18T16:10:55.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.482+0000 7f09f6166700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f09f00bf200 0x7f09f00bf5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:55.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.482+0000 7f09f6166700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f09f00bcfb0 0x7f09f01511f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:55.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.482+0000 7f09f6166700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09f00be390 con 0x7f09f00bfb90 2023-10-18T16:10:55.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.482+0000 7f09f6166700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09f01516e0 con 0x7f09f00bcfb0 2023-10-18T16:10:55.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.482+0000 7f09f6166700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09f00be820 con 0x7f09f00bf200 2023-10-18T16:10:55.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.482+0000 7f09eeffd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f09f00bcfb0 0x7f09f01511f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:55.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.482+0000 7f09effff700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f09f00bfb90 0x7f09f00691a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:48582/0 (socket says 172.21.15.88:48582) 2023-10-18T16:10:55.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.482+0000 7f09effff700 1 -- 172.21.15.88:0/3863358108 learned_addr learned my addr 172.21.15.88:0/3863358108 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:10:55.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.482+0000 7f09ef7fe700 1 --2- 172.21.15.88:0/3863358108 >> v2:172.21.15.151:3300/0 conn(0x7f09f00bf200 0x7f09f00bf5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:55.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.482+0000 7f09eeffd700 1 -- 172.21.15.88:0/3863358108 >> v2:172.21.15.151:3300/0 conn(0x7f09f00bf200 msgr2=0x7f09f00bf5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:10:55.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.482+0000 7f09eeffd700 1 --2- 172.21.15.88:0/3863358108 >> v2:172.21.15.151:3300/0 conn(0x7f09f00bf200 0x7f09f00bf5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:55.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.482+0000 7f09eeffd700 1 -- 172.21.15.88:0/3863358108 >> v1:172.21.15.88:6789/0 conn(0x7f09f00bfb90 legacy=0x7f09f00691a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:55.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.482+0000 7f09ee7fc700 1 -- 172.21.15.88:0/3863358108 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3799914166 0 0) 0x7f09f00be390 con 0x7f09f00bfb90 2023-10-18T16:10:55.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.482+0000 7f09eeffd700 1 -- 172.21.15.88:0/3863358108 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09f00698b0 con 0x7f09f00bcfb0 2023-10-18T16:10:55.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.482+0000 7f09eeffd700 1 --2- 172.21.15.88:0/3863358108 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09f00bcfb0 0x7f09f01511f0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f09f0068870 tx=0x7f09e0009060).ready entity=mon.0 client_cookie=e393c9ce6902ca0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:10:55.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.482+0000 7f09ee7fc700 1 -- 172.21.15.88:0/3863358108 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09e000b610 con 0x7f09f00bcfb0 2023-10-18T16:10:55.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.482+0000 7f09ee7fc700 1 -- 172.21.15.88:0/3863358108 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09e000b770 con 0x7f09f00bcfb0 2023-10-18T16:10:55.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.482+0000 7f09ee7fc700 1 -- 172.21.15.88:0/3863358108 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09e000baf0 con 0x7f09f00bcfb0 2023-10-18T16:10:55.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.483+0000 7f09f6166700 1 -- 172.21.15.88:0/3863358108 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09f00bcfb0 msgr2=0x7f09f01511f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:55.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.483+0000 7f09f6166700 1 --2- 172.21.15.88:0/3863358108 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09f00bcfb0 0x7f09f01511f0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f09f0068870 tx=0x7f09e0009060).stop 2023-10-18T16:10:55.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.483+0000 7f09f6166700 1 -- 172.21.15.88:0/3863358108 shutdown_connections 2023-10-18T16:10:55.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.483+0000 7f09f6166700 1 --2- 172.21.15.88:0/3863358108 >> v2:172.21.15.151:3300/0 conn(0x7f09f00bf200 0x7f09f00bf5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:55.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.483+0000 7f09f6166700 1 --2- 172.21.15.88:0/3863358108 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09f00bcfb0 0x7f09f01511f0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:55.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.483+0000 7f09f6166700 1 -- 172.21.15.88:0/3863358108 >> 172.21.15.88:0/3863358108 conn(0x7f09f0062650 msgr2=0x7f09f0062a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:10:55.492 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.483+0000 7f09f6166700 1 -- 172.21.15.88:0/3863358108 shutdown_connections 2023-10-18T16:10:55.492 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.483+0000 7f09f6166700 1 -- 172.21.15.88:0/3863358108 wait complete. 2023-10-18T16:10:55.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.485+0000 7f09f6166700 1 Processor -- start 2023-10-18T16:10:55.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.485+0000 7f09f6166700 1 -- start start 2023-10-18T16:10:55.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.485+0000 7f09f6166700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f09f00bcfb0 0x7f09f0161770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:55.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.485+0000 7f09f6166700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f09f00bf200 0x7f09f0141a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:55.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.485+0000 7f09f6166700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09f00bfb90 0x7f09f0141f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:55.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.485+0000 7f09f6166700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09f0156fb0 con 0x7f09f00bfb90 2023-10-18T16:10:55.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.485+0000 7f09f6166700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09f0157440 con 0x7f09f00bcfb0 2023-10-18T16:10:55.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.485+0000 7f09f6166700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09f01560c0 con 0x7f09f00bf200 2023-10-18T16:10:55.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.485+0000 7f09effff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09f00bfb90 0x7f09f0141f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:55.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.485+0000 7f09effff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09f00bfb90 0x7f09f0141f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41254/0 (socket says 172.21.15.88:41254) 2023-10-18T16:10:55.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.485+0000 7f09effff700 1 -- 172.21.15.88:0/3614897904 learned_addr learned my addr 172.21.15.88:0/3614897904 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:10:55.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.486+0000 7f09ef7fe700 1 --2- 172.21.15.88:0/3614897904 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f09f00bcfb0 0x7f09f0161770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:55.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.486+0000 7f09eeffd700 1 --2- 172.21.15.88:0/3614897904 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f09f00bf200 0x7f09f0141a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:55.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.486+0000 7f09effff700 1 -- 172.21.15.88:0/3614897904 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f09f00bf200 msgr2=0x7f09f0141a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:55.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.486+0000 7f09effff700 1 --2- 172.21.15.88:0/3614897904 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f09f00bf200 0x7f09f0141a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:55.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.486+0000 7f09effff700 1 -- 172.21.15.88:0/3614897904 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f09f00bcfb0 msgr2=0x7f09f0161770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:55.498 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.486+0000 7f09effff700 1 --2- 172.21.15.88:0/3614897904 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f09f00bcfb0 0x7f09f0161770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:55.498 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.486+0000 7f09effff700 1 -- 172.21.15.88:0/3614897904 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09e00092a0 con 0x7f09f00bfb90 2023-10-18T16:10:55.498 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.486+0000 7f09eeffd700 1 --2- 172.21.15.88:0/3614897904 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f09f00bf200 0x7f09f0141a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-10-18T16:10:55.498 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.486+0000 7f09ef7fe700 1 --2- 172.21.15.88:0/3614897904 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f09f00bcfb0 0x7f09f0161770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:10:55.498 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.486+0000 7f09effff700 1 --2- 172.21.15.88:0/3614897904 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09f00bfb90 0x7f09f0141f50 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f09e800c7e0 tx=0x7f09e8007f90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:10:55.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.486+0000 7f09d77fe700 1 -- 172.21.15.88:0/3614897904 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09e8005d30 con 0x7f09f00bfb90 2023-10-18T16:10:55.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.486+0000 7f09d77fe700 1 -- 172.21.15.88:0/3614897904 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09e8005e90 con 0x7f09f00bfb90 2023-10-18T16:10:55.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.486+0000 7f09f6166700 1 -- 172.21.15.88:0/3614897904 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09f01563a0 con 0x7f09f00bfb90 2023-10-18T16:10:55.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.486+0000 7f09d77fe700 1 -- 172.21.15.88:0/3614897904 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09e8007320 con 0x7f09f00bfb90 2023-10-18T16:10:55.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.486+0000 7f09f6166700 1 -- 172.21.15.88:0/3614897904 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09f0156960 con 0x7f09f00bfb90 2023-10-18T16:10:55.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.487+0000 7f09f6166700 1 -- 172.21.15.88:0/3614897904 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09f0157120 con 0x7f09f00bfb90 2023-10-18T16:10:55.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.487+0000 7f09d77fe700 1 -- 172.21.15.88:0/3614897904 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f09e8007710 con 0x7f09f00bfb90 2023-10-18T16:10:55.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.488+0000 7f09d77fe700 1 --2- 172.21.15.88:0/3614897904 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f09dc05f120 0x7f09dc0615e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:55.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.488+0000 7f09d77fe700 1 -- 172.21.15.88:0/3614897904 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6725+0+0 (secure 0 0 0) 0x7f09e807a520 con 0x7f09f00bfb90 2023-10-18T16:10:55.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.488+0000 7f09ef7fe700 1 --2- 172.21.15.88:0/3614897904 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f09dc05f120 0x7f09dc0615e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:55.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.488+0000 7f09ef7fe700 1 --2- 172.21.15.88:0/3614897904 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f09dc05f120 0x7f09dc0615e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f09e400bb60 tx=0x7f09e4008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:10:55.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.492+0000 7f09d77fe700 1 -- 172.21.15.88:0/3614897904 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f09e804b590 con 0x7f09f00bfb90 2023-10-18T16:10:55.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.691+0000 7f09f6166700 1 -- 172.21.15.88:0/3614897904 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f09f0063bf0 con 0x7f09f00bfb90 2023-10-18T16:10:55.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.691+0000 7f09d77fe700 1 -- 172.21.15.88:0/3614897904 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7f09e804ebb0 con 0x7f09f00bfb90 2023-10-18T16:10:55.696 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:10:55.697 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.693+0000 7f09f6166700 1 -- 172.21.15.88:0/3614897904 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f09dc05f120 msgr2=0x7f09dc0615e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:55.697 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.693+0000 7f09f6166700 1 --2- 172.21.15.88:0/3614897904 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f09dc05f120 0x7f09dc0615e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f09e400bb60 tx=0x7f09e4008040).stop 2023-10-18T16:10:55.697 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.693+0000 7f09f6166700 1 -- 172.21.15.88:0/3614897904 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09f00bfb90 msgr2=0x7f09f0141f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:55.698 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.693+0000 7f09f6166700 1 --2- 172.21.15.88:0/3614897904 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09f00bfb90 0x7f09f0141f50 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f09e800c7e0 tx=0x7f09e8007f90).stop 2023-10-18T16:10:55.698 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.693+0000 7f09f6166700 1 -- 172.21.15.88:0/3614897904 shutdown_connections 2023-10-18T16:10:55.698 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.693+0000 7f09f6166700 1 --2- 172.21.15.88:0/3614897904 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f09dc05f120 0x7f09dc0615e0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:55.698 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.693+0000 7f09f6166700 1 --2- 172.21.15.88:0/3614897904 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09f00bfb90 0x7f09f0141f50 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:55.698 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.693+0000 7f09f6166700 1 --2- 172.21.15.88:0/3614897904 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f09f00bcfb0 0x7f09f0161770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:55.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.693+0000 7f09f6166700 1 --2- 172.21.15.88:0/3614897904 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f09f00bf200 0x7f09f0141a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:55.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.693+0000 7f09f6166700 1 -- 172.21.15.88:0/3614897904 >> 172.21.15.88:0/3614897904 conn(0x7f09f0062650 msgr2=0x7f09f00660f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:10:55.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.693+0000 7f09f6166700 1 -- 172.21.15.88:0/3614897904 shutdown_connections 2023-10-18T16:10:55.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:55.693+0000 7f09f6166700 1 -- 172.21.15.88:0/3614897904 wait complete. 2023-10-18T16:10:55.709 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:10:57.932 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~82s 2023-10-18T16:10:57.932 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~95s 2023-10-18T16:10:57.932 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~128s 2023-10-18T16:10:57.932 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~108s 2023-10-18T16:10:57.933 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~189s 2023-10-18T16:10:57.933 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.f is failed for ~141s 2023-10-18T16:10:58.710 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:10:58.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:58.833+0000 7f9f33ff3700 1 Processor -- start 2023-10-18T16:10:58.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:58.834+0000 7f9f33ff3700 1 -- start start 2023-10-18T16:10:58.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:58.834+0000 7f9f33ff3700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9f2c0c5660 0x7f9f2c151cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:58.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:58.834+0000 7f9f33ff3700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f9f2c0bf400 0x7f9f2c1521e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:58.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:58.834+0000 7f9f33ff3700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f2c067dd0 con 0x7f9f2c151910 2023-10-18T16:10:58.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:58.834+0000 7f9f33ff3700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f2c067b20 con 0x7f9f2c0c5660 2023-10-18T16:10:58.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:58.834+0000 7f9f33ff3700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f2c0b9b60 con 0x7f9f2c0bf400 2023-10-18T16:10:58.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:58.834+0000 7f9f3158e700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9f2c0c5660 0x7f9f2c151cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:58.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:58.835+0000 7f9f3158e700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9f2c0c5660 0x7f9f2c151cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:58844/0 (socket says 172.21.15.88:58844) 2023-10-18T16:10:58.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:58.835+0000 7f9f3158e700 1 -- 172.21.15.88:0/1313626784 learned_addr learned my addr 172.21.15.88:0/1313626784 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:10:58.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:58.835+0000 7f9f32590700 1 --2- 172.21.15.88:0/1313626784 >> v2:172.21.15.151:3300/0 conn(0x7f9f2c0bf400 0x7f9f2c1521e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:58.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:58.835+0000 7f9f3158e700 1 -- 172.21.15.88:0/1313626784 >> v2:172.21.15.151:3300/0 conn(0x7f9f2c0bf400 msgr2=0x7f9f2c1521e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:10:58.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:58.835+0000 7f9f3158e700 1 --2- 172.21.15.88:0/1313626784 >> v2:172.21.15.151:3300/0 conn(0x7f9f2c0bf400 0x7f9f2c1521e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:58.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:58.835+0000 7f9f3158e700 1 -- 172.21.15.88:0/1313626784 >> v1:172.21.15.151:6789/0 conn(0x7f9f2c151910 legacy=0x7f9f2c068650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:10:58.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:58.835+0000 7f9f3158e700 1 -- 172.21.15.88:0/1313626784 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f2c0b9ff0 con 0x7f9f2c0c5660 2023-10-18T16:10:58.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:58.835+0000 7f9f3158e700 1 --2- 172.21.15.88:0/1313626784 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f2c0c5660 0x7f9f2c151cf0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f9f2c06ac10 tx=0x7f9f1c0090f0).ready entity=mon.0 client_cookie=fdfd0a2de75e51f6 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:10:58.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:58.835+0000 7f9f30d8d700 1 -- 172.21.15.88:0/1313626784 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f1c00b630 con 0x7f9f2c0c5660 2023-10-18T16:10:58.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:58.835+0000 7f9f30d8d700 1 -- 172.21.15.88:0/1313626784 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f1c00b790 con 0x7f9f2c0c5660 2023-10-18T16:10:58.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:58.835+0000 7f9f30d8d700 1 -- 172.21.15.88:0/1313626784 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f1c00bb10 con 0x7f9f2c0c5660 2023-10-18T16:10:58.844 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:58.836+0000 7f9f33ff3700 1 -- 172.21.15.88:0/1313626784 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f2c0c5660 msgr2=0x7f9f2c151cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:58.844 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:58.836+0000 7f9f33ff3700 1 --2- 172.21.15.88:0/1313626784 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f2c0c5660 0x7f9f2c151cf0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f9f2c06ac10 tx=0x7f9f1c0090f0).stop 2023-10-18T16:10:58.844 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:58.836+0000 7f9f33ff3700 1 -- 172.21.15.88:0/1313626784 shutdown_connections 2023-10-18T16:10:58.844 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:58.836+0000 7f9f33ff3700 1 --2- 172.21.15.88:0/1313626784 >> v2:172.21.15.151:3300/0 conn(0x7f9f2c0bf400 0x7f9f2c1521e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:58.844 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:58.836+0000 7f9f33ff3700 1 --2- 172.21.15.88:0/1313626784 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f2c0c5660 0x7f9f2c151cf0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:58.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:58.836+0000 7f9f33ff3700 1 -- 172.21.15.88:0/1313626784 >> 172.21.15.88:0/1313626784 conn(0x7f9f2c0c4940 msgr2=0x7f9f2c0c4d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:10:58.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:58.836+0000 7f9f33ff3700 1 -- 172.21.15.88:0/1313626784 shutdown_connections 2023-10-18T16:10:58.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:58.836+0000 7f9f33ff3700 1 -- 172.21.15.88:0/1313626784 wait complete. 2023-10-18T16:10:58.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:58.837+0000 7f9f33ff3700 1 Processor -- start 2023-10-18T16:10:58.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:58.837+0000 7f9f33ff3700 1 -- start start 2023-10-18T16:10:58.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:58.838+0000 7f9f33ff3700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f2c0bf400 0x7f9f2c15ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:58.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:58.838+0000 7f9f33ff3700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9f2c0c5660 0x7f9f2c0bbaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:58.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:58.838+0000 7f9f33ff3700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9f2c151910 0x7f9f2c1507e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:58.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:58.838+0000 7f9f33ff3700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f2c0bbf90 con 0x7f9f2c0bf400 2023-10-18T16:10:58.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:58.838+0000 7f9f33ff3700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f2c14f4c0 con 0x7f9f2c0c5660 2023-10-18T16:10:58.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:58.838+0000 7f9f33ff3700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f2c15e7c0 con 0x7f9f2c151910 2023-10-18T16:10:58.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:58.838+0000 7f9f31d8f700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f2c0bf400 0x7f9f2c15ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:58.850 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:58.838+0000 7f9f31d8f700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f2c0bf400 0x7f9f2c15ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:58856/0 (socket says 172.21.15.88:58856) 2023-10-18T16:10:58.850 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:58.838+0000 7f9f31d8f700 1 -- 172.21.15.88:0/1840682707 learned_addr learned my addr 172.21.15.88:0/1840682707 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:10:58.850 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:58.838+0000 7f9f3158e700 1 --2- 172.21.15.88:0/1840682707 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9f2c0c5660 0x7f9f2c0bbaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:58.850 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:58.838+0000 7f9f32590700 1 --2- 172.21.15.88:0/1840682707 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9f2c151910 0x7f9f2c1507e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:58.850 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:58.838+0000 7f9f31d8f700 1 -- 172.21.15.88:0/1840682707 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9f2c151910 msgr2=0x7f9f2c1507e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:58.851 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:58.838+0000 7f9f31d8f700 1 --2- 172.21.15.88:0/1840682707 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9f2c151910 0x7f9f2c1507e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:58.851 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:58.838+0000 7f9f31d8f700 1 -- 172.21.15.88:0/1840682707 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9f2c0c5660 msgr2=0x7f9f2c0bbaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:58.851 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:58.838+0000 7f9f31d8f700 1 --2- 172.21.15.88:0/1840682707 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9f2c0c5660 0x7f9f2c0bbaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:58.851 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:58.838+0000 7f9f31d8f700 1 -- 172.21.15.88:0/1840682707 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f1c009370 con 0x7f9f2c0bf400 2023-10-18T16:10:58.851 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:58.838+0000 7f9f3158e700 1 --2- 172.21.15.88:0/1840682707 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9f2c0c5660 0x7f9f2c0bbaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:10:58.851 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:58.838+0000 7f9f31d8f700 1 --2- 172.21.15.88:0/1840682707 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f2c0bf400 0x7f9f2c15ec50 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f9f2800b490 tx=0x7f9f28006f20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:10:58.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:58.838+0000 7f9f21ffb700 1 -- 172.21.15.88:0/1840682707 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f28012410 con 0x7f9f2c0bf400 2023-10-18T16:10:58.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:58.838+0000 7f9f21ffb700 1 -- 172.21.15.88:0/1840682707 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f28012590 con 0x7f9f2c0bf400 2023-10-18T16:10:58.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:58.838+0000 7f9f33ff3700 1 -- 172.21.15.88:0/1840682707 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f2c14e9a0 con 0x7f9f2c0bf400 2023-10-18T16:10:58.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:58.838+0000 7f9f21ffb700 1 -- 172.21.15.88:0/1840682707 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f28012950 con 0x7f9f2c0bf400 2023-10-18T16:10:58.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:58.839+0000 7f9f33ff3700 1 -- 172.21.15.88:0/1840682707 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f2c15e170 con 0x7f9f2c0bf400 2023-10-18T16:10:58.853 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:58.840+0000 7f9f21ffb700 1 -- 172.21.15.88:0/1840682707 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f9f28012d00 con 0x7f9f2c0bf400 2023-10-18T16:10:58.853 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:58.840+0000 7f9f33ff3700 1 -- 172.21.15.88:0/1840682707 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f2c06c4e0 con 0x7f9f2c0bf400 2023-10-18T16:10:58.853 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:58.840+0000 7f9f21ffb700 1 --2- 172.21.15.88:0/1840682707 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f9f1005f180 0x7f9f10061640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:10:58.853 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:58.840+0000 7f9f21ffb700 1 -- 172.21.15.88:0/1840682707 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6725+0+0 (secure 0 0 0) 0x7f9f2807a280 con 0x7f9f2c0bf400 2023-10-18T16:10:58.853 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:58.840+0000 7f9f3158e700 1 --2- 172.21.15.88:0/1840682707 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f9f1005f180 0x7f9f10061640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:10:58.854 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:58.841+0000 7f9f3158e700 1 --2- 172.21.15.88:0/1840682707 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f9f1005f180 0x7f9f10061640 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f9f1c009130 tx=0x7f9f1c015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:10:58.854 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:58.844+0000 7f9f21ffb700 1 -- 172.21.15.88:0/1840682707 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f9f2804b2f0 con 0x7f9f2c0bf400 2023-10-18T16:10:59.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:59.041+0000 7f9f33ff3700 1 -- 172.21.15.88:0/1840682707 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f9f2c0bb740 con 0x7f9f2c0bf400 2023-10-18T16:10:59.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:59.041+0000 7f9f21ffb700 1 -- 172.21.15.88:0/1840682707 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7f9f2804e910 con 0x7f9f2c0bf400 2023-10-18T16:10:59.046 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:10:59.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:59.043+0000 7f9f33ff3700 1 -- 172.21.15.88:0/1840682707 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f9f1005f180 msgr2=0x7f9f10061640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:59.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:59.043+0000 7f9f33ff3700 1 --2- 172.21.15.88:0/1840682707 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f9f1005f180 0x7f9f10061640 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f9f1c009130 tx=0x7f9f1c015000).stop 2023-10-18T16:10:59.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:59.043+0000 7f9f33ff3700 1 -- 172.21.15.88:0/1840682707 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f2c0bf400 msgr2=0x7f9f2c15ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:10:59.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:59.043+0000 7f9f33ff3700 1 --2- 172.21.15.88:0/1840682707 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f2c0bf400 0x7f9f2c15ec50 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f9f2800b490 tx=0x7f9f28006f20).stop 2023-10-18T16:10:59.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:59.043+0000 7f9f33ff3700 1 -- 172.21.15.88:0/1840682707 shutdown_connections 2023-10-18T16:10:59.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:59.043+0000 7f9f33ff3700 1 --2- 172.21.15.88:0/1840682707 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f9f1005f180 0x7f9f10061640 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:59.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:59.043+0000 7f9f33ff3700 1 --2- 172.21.15.88:0/1840682707 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9f2c151910 0x7f9f2c1507e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:59.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:59.043+0000 7f9f33ff3700 1 --2- 172.21.15.88:0/1840682707 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f2c0bf400 0x7f9f2c15ec50 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:59.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:59.043+0000 7f9f33ff3700 1 --2- 172.21.15.88:0/1840682707 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9f2c0c5660 0x7f9f2c0bbaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:10:59.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:59.043+0000 7f9f33ff3700 1 -- 172.21.15.88:0/1840682707 >> 172.21.15.88:0/1840682707 conn(0x7f9f2c0c4940 msgr2=0x7f9f2c14b4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:10:59.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:59.043+0000 7f9f33ff3700 1 -- 172.21.15.88:0/1840682707 shutdown_connections 2023-10-18T16:10:59.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:10:59.043+0000 7f9f33ff3700 1 -- 172.21.15.88:0/1840682707 wait complete. 2023-10-18T16:10:59.058 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:11:02.059 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:11:02.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.509+0000 7f09538d2700 1 Processor -- start 2023-10-18T16:11:02.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.509+0000 7f09538d2700 1 -- start start 2023-10-18T16:11:02.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.510+0000 7f09538d2700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f094c0c5660 0x7f094c151cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:02.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.510+0000 7f09538d2700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f094c0bf400 0x7f094c1521e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:02.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.510+0000 7f09538d2700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f094c067dd0 con 0x7f094c151910 2023-10-18T16:11:02.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.510+0000 7f09538d2700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f094c067b20 con 0x7f094c0c5660 2023-10-18T16:11:02.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.510+0000 7f09538d2700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f094c0b9b60 con 0x7f094c0bf400 2023-10-18T16:11:02.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.510+0000 7f0950e6d700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f094c0c5660 0x7f094c151cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:02.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.510+0000 7f0950e6d700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f094c0c5660 0x7f094c151cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:58864/0 (socket says 172.21.15.88:58864) 2023-10-18T16:11:02.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.510+0000 7f0950e6d700 1 -- 172.21.15.88:0/4212973770 learned_addr learned my addr 172.21.15.88:0/4212973770 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:11:02.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.510+0000 7f0951e6f700 1 --2- 172.21.15.88:0/4212973770 >> v2:172.21.15.151:3300/0 conn(0x7f094c0bf400 0x7f094c1521e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:02.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.510+0000 7f0950e6d700 1 -- 172.21.15.88:0/4212973770 >> v2:172.21.15.151:3300/0 conn(0x7f094c0bf400 msgr2=0x7f094c1521e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:11:02.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.510+0000 7f0950e6d700 1 --2- 172.21.15.88:0/4212973770 >> v2:172.21.15.151:3300/0 conn(0x7f094c0bf400 0x7f094c1521e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:02.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.510+0000 7f0950e6d700 1 -- 172.21.15.88:0/4212973770 >> v1:172.21.15.151:6789/0 conn(0x7f094c151910 legacy=0x7f094c068650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:02.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.510+0000 7f0950e6d700 1 -- 172.21.15.88:0/4212973770 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f094c0b9ff0 con 0x7f094c0c5660 2023-10-18T16:11:02.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.510+0000 7f0951e6f700 1 --2- 172.21.15.88:0/4212973770 >> v2:172.21.15.151:3300/0 conn(0x7f094c0bf400 0x7f094c1521e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:11:02.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.511+0000 7f0950e6d700 1 --2- 172.21.15.88:0/4212973770 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f094c0c5660 0x7f094c151cf0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f094c06ac10 tx=0x7f09400090f0).ready entity=mon.0 client_cookie=bb5ea691189b67f2 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:11:02.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.511+0000 7f094bfff700 1 -- 172.21.15.88:0/4212973770 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f094000b690 con 0x7f094c0c5660 2023-10-18T16:11:02.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.511+0000 7f094bfff700 1 -- 172.21.15.88:0/4212973770 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f094000b7f0 con 0x7f094c0c5660 2023-10-18T16:11:02.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.511+0000 7f094bfff700 1 -- 172.21.15.88:0/4212973770 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f094000bb70 con 0x7f094c0c5660 2023-10-18T16:11:02.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.511+0000 7f09538d2700 1 -- 172.21.15.88:0/4212973770 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f094c0c5660 msgr2=0x7f094c151cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:02.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.511+0000 7f09538d2700 1 --2- 172.21.15.88:0/4212973770 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f094c0c5660 0x7f094c151cf0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f094c06ac10 tx=0x7f09400090f0).stop 2023-10-18T16:11:02.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.512+0000 7f09538d2700 1 -- 172.21.15.88:0/4212973770 shutdown_connections 2023-10-18T16:11:02.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.512+0000 7f09538d2700 1 --2- 172.21.15.88:0/4212973770 >> v2:172.21.15.151:3300/0 conn(0x7f094c0bf400 0x7f094c1521e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:02.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.512+0000 7f09538d2700 1 --2- 172.21.15.88:0/4212973770 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f094c0c5660 0x7f094c151cf0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:02.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.512+0000 7f09538d2700 1 -- 172.21.15.88:0/4212973770 >> 172.21.15.88:0/4212973770 conn(0x7f094c0c4940 msgr2=0x7f094c0c4d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:11:02.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.512+0000 7f09538d2700 1 -- 172.21.15.88:0/4212973770 shutdown_connections 2023-10-18T16:11:02.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.512+0000 7f09538d2700 1 -- 172.21.15.88:0/4212973770 wait complete. 2023-10-18T16:11:02.530 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.513+0000 7f09538d2700 1 Processor -- start 2023-10-18T16:11:02.530 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.513+0000 7f09538d2700 1 -- start start 2023-10-18T16:11:02.530 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.514+0000 7f09538d2700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f094c0bf400 0x7f094c0687f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:02.530 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.514+0000 7f09538d2700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f094c0c5660 0x7f094c145830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:02.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.514+0000 7f09538d2700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f094c151910 0x7f094c145d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:02.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.514+0000 7f09538d2700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f094c162660 con 0x7f094c0c5660 2023-10-18T16:11:02.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.514+0000 7f09538d2700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f094c1453a0 con 0x7f094c151910 2023-10-18T16:11:02.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.514+0000 7f09538d2700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f094c146210 con 0x7f094c0bf400 2023-10-18T16:11:02.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.514+0000 7f0950e6d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f094c0c5660 0x7f094c145830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:02.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.514+0000 7f0950e6d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f094c0c5660 0x7f094c145830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:58866/0 (socket says 172.21.15.88:58866) 2023-10-18T16:11:02.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.514+0000 7f0950e6d700 1 -- 172.21.15.88:0/3439759121 learned_addr learned my addr 172.21.15.88:0/3439759121 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:11:02.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.514+0000 7f095166e700 1 --2- 172.21.15.88:0/3439759121 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f094c0bf400 0x7f094c0687f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:02.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.514+0000 7f0951e6f700 1 --2- 172.21.15.88:0/3439759121 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f094c151910 0x7f094c145d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:02.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.514+0000 7f0950e6d700 1 -- 172.21.15.88:0/3439759121 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f094c0bf400 msgr2=0x7f094c0687f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:02.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.514+0000 7f0950e6d700 1 --2- 172.21.15.88:0/3439759121 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f094c0bf400 0x7f094c0687f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:02.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.514+0000 7f0950e6d700 1 -- 172.21.15.88:0/3439759121 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f094c151910 msgr2=0x7f094c145d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:02.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.514+0000 7f0950e6d700 1 --2- 172.21.15.88:0/3439759121 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f094c151910 0x7f094c145d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:02.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.514+0000 7f0950e6d700 1 -- 172.21.15.88:0/3439759121 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0940009370 con 0x7f094c0c5660 2023-10-18T16:11:02.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.514+0000 7f0951e6f700 1 --2- 172.21.15.88:0/3439759121 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f094c151910 0x7f094c145d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:11:02.534 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.514+0000 7f0950e6d700 1 --2- 172.21.15.88:0/3439759121 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f094c0c5660 0x7f094c145830 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f0940009fd0 tx=0x7f0940017000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:11:02.534 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.514+0000 7f09497fa700 1 -- 172.21.15.88:0/3439759121 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f094000b960 con 0x7f094c0c5660 2023-10-18T16:11:02.534 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.514+0000 7f09497fa700 1 -- 172.21.15.88:0/3439759121 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f094000bac0 con 0x7f094c0c5660 2023-10-18T16:11:02.534 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.514+0000 7f09538d2700 1 -- 172.21.15.88:0/3439759121 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f094c1620f0 con 0x7f094c0c5660 2023-10-18T16:11:02.534 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.514+0000 7f09497fa700 1 -- 172.21.15.88:0/3439759121 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f094001b660 con 0x7f094c0c5660 2023-10-18T16:11:02.535 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.514+0000 7f09538d2700 1 -- 172.21.15.88:0/3439759121 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f094c162380 con 0x7f094c0c5660 2023-10-18T16:11:02.535 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.515+0000 7f09538d2700 1 -- 172.21.15.88:0/3439759121 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f094c141e40 con 0x7f094c0c5660 2023-10-18T16:11:02.535 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.516+0000 7f09497fa700 1 -- 172.21.15.88:0/3439759121 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f094001b9d0 con 0x7f094c0c5660 2023-10-18T16:11:02.535 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.516+0000 7f09497fa700 1 --2- 172.21.15.88:0/3439759121 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f093805f180 0x7f0938061640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:02.535 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.516+0000 7f09497fa700 1 -- 172.21.15.88:0/3439759121 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6725+0+0 (secure 0 0 0) 0x7f094007bc10 con 0x7f094c0c5660 2023-10-18T16:11:02.536 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.516+0000 7f095166e700 1 --2- 172.21.15.88:0/3439759121 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f093805f180 0x7f0938061640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:02.536 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.517+0000 7f095166e700 1 --2- 172.21.15.88:0/3439759121 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f093805f180 0x7f0938061640 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f0944003860 tx=0x7f0944007d40).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:11:02.536 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.520+0000 7f09497fa700 1 -- 172.21.15.88:0/3439759121 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f094c141e40 con 0x7f094c0c5660 2023-10-18T16:11:02.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.719+0000 7f09538d2700 1 -- 172.21.15.88:0/3439759121 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f094c141e40 con 0x7f094c0c5660 2023-10-18T16:11:02.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.719+0000 7f09497fa700 1 -- 172.21.15.88:0/3439759121 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7f094c141e40 con 0x7f094c0c5660 2023-10-18T16:11:02.723 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:11:02.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.720+0000 7f09538d2700 1 -- 172.21.15.88:0/3439759121 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f093805f180 msgr2=0x7f0938061640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:02.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.720+0000 7f09538d2700 1 --2- 172.21.15.88:0/3439759121 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f093805f180 0x7f0938061640 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f0944003860 tx=0x7f0944007d40).stop 2023-10-18T16:11:02.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.720+0000 7f09538d2700 1 -- 172.21.15.88:0/3439759121 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f094c0c5660 msgr2=0x7f094c145830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:02.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.720+0000 7f09538d2700 1 --2- 172.21.15.88:0/3439759121 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f094c0c5660 0x7f094c145830 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f0940009fd0 tx=0x7f0940017000).stop 2023-10-18T16:11:02.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.721+0000 7f09538d2700 1 -- 172.21.15.88:0/3439759121 shutdown_connections 2023-10-18T16:11:02.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.721+0000 7f09538d2700 1 --2- 172.21.15.88:0/3439759121 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f093805f180 0x7f0938061640 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:02.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.721+0000 7f09538d2700 1 --2- 172.21.15.88:0/3439759121 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f094c151910 0x7f094c145d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:02.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.721+0000 7f09538d2700 1 --2- 172.21.15.88:0/3439759121 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f094c0bf400 0x7f094c0687f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:02.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.721+0000 7f09538d2700 1 --2- 172.21.15.88:0/3439759121 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f094c0c5660 0x7f094c145830 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:02.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.721+0000 7f09538d2700 1 -- 172.21.15.88:0/3439759121 >> 172.21.15.88:0/3439759121 conn(0x7f094c0c4940 msgr2=0x7f094c069260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:11:02.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.721+0000 7f09538d2700 1 -- 172.21.15.88:0/3439759121 shutdown_connections 2023-10-18T16:11:02.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:02.721+0000 7f09538d2700 1 -- 172.21.15.88:0/3439759121 wait complete. 2023-10-18T16:11:02.736 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:11:04.242 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~88s 2023-10-18T16:11:04.243 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~101s 2023-10-18T16:11:04.243 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~134s 2023-10-18T16:11:04.243 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~114s 2023-10-18T16:11:04.243 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~195s 2023-10-18T16:11:04.243 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.f is failed for ~147s 2023-10-18T16:11:05.737 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:11:05.863 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:05.860+0000 7fbc0e63a700 1 Processor -- start 2023-10-18T16:11:05.864 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:05.860+0000 7fbc0e63a700 1 -- start start 2023-10-18T16:11:05.866 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:05.861+0000 7fbc0e63a700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbc0814d8c0 0x7fbc08064440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:05.866 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:05.861+0000 7fbc0e63a700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc08064150 con 0x7fbc08065360 2023-10-18T16:11:05.866 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:05.861+0000 7fbc0e63a700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc08153a40 con 0x7fbc080a9050 2023-10-18T16:11:05.866 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:05.861+0000 7fbc0e63a700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc08063e70 con 0x7fbc0814d8c0 2023-10-18T16:11:05.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:05.861+0000 7fbc0cbd7700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fbc08065360 0x7fbc0815acb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:58838/0 (socket says 172.21.15.88:58838) 2023-10-18T16:11:05.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:05.861+0000 7fbc0cbd7700 1 -- 172.21.15.88:0/1594778039 learned_addr learned my addr 172.21.15.88:0/1594778039 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:11:05.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:05.861+0000 7fbc07fff700 1 --2- 172.21.15.88:0/1594778039 >> v2:172.21.15.88:3300/0 conn(0x7fbc0814d8c0 0x7fbc08064440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:05.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:05.861+0000 7fbc07fff700 1 -- 172.21.15.88:0/1594778039 >> v1:172.21.15.151:6789/0 conn(0x7fbc080a9050 legacy=0x7fbc08153ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:11:05.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:05.861+0000 7fbc07fff700 1 -- 172.21.15.88:0/1594778039 >> v1:172.21.15.88:6789/0 conn(0x7fbc08065360 legacy=0x7fbc0815acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:05.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:05.861+0000 7fbc07fff700 1 -- 172.21.15.88:0/1594778039 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc080c72c0 con 0x7fbc0814d8c0 2023-10-18T16:11:05.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:05.861+0000 7fbc06ffd700 1 -- 172.21.15.88:0/1594778039 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3316390093 0 0) 0x7fbc08064150 con 0x7fbc08065360 2023-10-18T16:11:05.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:05.861+0000 7fbc07fff700 1 --2- 172.21.15.88:0/1594778039 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc0814d8c0 0x7fbc08064440 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7fbbf8009760 tx=0x7fbbf8009060).ready entity=mon.0 client_cookie=622563e000ec2bab server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:11:05.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:05.861+0000 7fbc06ffd700 1 -- 172.21.15.88:0/1594778039 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbf80090d0 con 0x7fbc0814d8c0 2023-10-18T16:11:05.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:05.862+0000 7fbc06ffd700 1 -- 172.21.15.88:0/1594778039 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbf800b520 con 0x7fbc0814d8c0 2023-10-18T16:11:05.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:05.862+0000 7fbc06ffd700 1 -- 172.21.15.88:0/1594778039 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbf800b8a0 con 0x7fbc0814d8c0 2023-10-18T16:11:05.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:05.862+0000 7fbc0e63a700 1 -- 172.21.15.88:0/1594778039 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc0814d8c0 msgr2=0x7fbc08064440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:05.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:05.862+0000 7fbc0e63a700 1 --2- 172.21.15.88:0/1594778039 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc0814d8c0 0x7fbc08064440 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7fbbf8009760 tx=0x7fbbf8009060).stop 2023-10-18T16:11:05.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:05.862+0000 7fbc0e63a700 1 -- 172.21.15.88:0/1594778039 shutdown_connections 2023-10-18T16:11:05.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:05.862+0000 7fbc0e63a700 1 --2- 172.21.15.88:0/1594778039 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc0814d8c0 0x7fbc08064440 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:05.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:05.862+0000 7fbc0e63a700 1 -- 172.21.15.88:0/1594778039 >> 172.21.15.88:0/1594778039 conn(0x7fbc0806a810 msgr2=0x7fbc080c6a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:11:05.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:05.862+0000 7fbc0e63a700 1 -- 172.21.15.88:0/1594778039 shutdown_connections 2023-10-18T16:11:05.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:05.862+0000 7fbc0e63a700 1 -- 172.21.15.88:0/1594778039 wait complete. 2023-10-18T16:11:05.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:05.863+0000 7fbc0e63a700 1 Processor -- start 2023-10-18T16:11:05.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:05.863+0000 7fbc0e63a700 1 -- start start 2023-10-18T16:11:05.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:05.864+0000 7fbc0e63a700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc08065360 0x7fbc080bb860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:05.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:05.864+0000 7fbc0e63a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbc080a9050 0x7fbc080aba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:05.874 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:05.864+0000 7fbc0e63a700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbc0814d8c0 0x7fbc080abf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:05.874 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:05.864+0000 7fbc0e63a700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc080ac440 con 0x7fbc08065360 2023-10-18T16:11:05.874 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:05.864+0000 7fbc0e63a700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc080ab2d0 con 0x7fbc080a9050 2023-10-18T16:11:05.874 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:05.864+0000 7fbc0e63a700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc080bc440 con 0x7fbc0814d8c0 2023-10-18T16:11:05.874 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:05.864+0000 7fbc07fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc08065360 0x7fbc080bb860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:05.875 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:05.864+0000 7fbc07fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc08065360 0x7fbc080bb860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:58886/0 (socket says 172.21.15.88:58886) 2023-10-18T16:11:05.875 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:05.864+0000 7fbc07fff700 1 -- 172.21.15.88:0/3075422168 learned_addr learned my addr 172.21.15.88:0/3075422168 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:11:05.875 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:05.864+0000 7fbc07fff700 1 -- 172.21.15.88:0/3075422168 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbc0814d8c0 msgr2=0x7fbc080abf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:05.875 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:05.864+0000 7fbc07fff700 1 --2- 172.21.15.88:0/3075422168 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbc0814d8c0 0x7fbc080abf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-10-18T16:11:05.875 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:05.864+0000 7fbc07fff700 1 -- 172.21.15.88:0/3075422168 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbc080a9050 msgr2=0x7fbc080aba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:05.876 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:05.864+0000 7fbc077fe700 1 --2- 172.21.15.88:0/3075422168 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbc080a9050 0x7fbc080aba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:05.876 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:05.865+0000 7fbc07fff700 1 --2- 172.21.15.88:0/3075422168 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbc080a9050 0x7fbc080aba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:05.876 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:05.865+0000 7fbc07fff700 1 -- 172.21.15.88:0/3075422168 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbf80092a0 con 0x7fbc08065360 2023-10-18T16:11:05.876 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:05.865+0000 7fbc07fff700 1 --2- 172.21.15.88:0/3075422168 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc08065360 0x7fbc080bb860 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7fbbf8009eb0 tx=0x7fbbf8017000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:11:05.876 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:05.865+0000 7fbbeffff700 1 -- 172.21.15.88:0/3075422168 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbf8011af0 con 0x7fbc08065360 2023-10-18T16:11:05.877 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:05.865+0000 7fbbeffff700 1 -- 172.21.15.88:0/3075422168 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbf8011c50 con 0x7fbc08065360 2023-10-18T16:11:05.877 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:05.865+0000 7fbc0e63a700 1 -- 172.21.15.88:0/3075422168 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc080c2be0 con 0x7fbc08065360 2023-10-18T16:11:05.877 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:05.865+0000 7fbbeffff700 1 -- 172.21.15.88:0/3075422168 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbf801b440 con 0x7fbc08065360 2023-10-18T16:11:05.877 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:05.865+0000 7fbc0e63a700 1 -- 172.21.15.88:0/3075422168 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc080bad40 con 0x7fbc08065360 2023-10-18T16:11:05.877 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:05.866+0000 7fbbeffff700 1 -- 172.21.15.88:0/3075422168 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7fbbf801b7f0 con 0x7fbc08065360 2023-10-18T16:11:05.877 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:05.866+0000 7fbc0e63a700 1 -- 172.21.15.88:0/3075422168 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc080ba3d0 con 0x7fbc08065360 2023-10-18T16:11:05.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:05.866+0000 7fbbeffff700 1 --2- 172.21.15.88:0/3075422168 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fbbf405f120 0x7fbbf40615e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:05.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:05.866+0000 7fbbeffff700 1 -- 172.21.15.88:0/3075422168 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6725+0+0 (secure 0 0 0) 0x7fbbf807bd40 con 0x7fbc08065360 2023-10-18T16:11:05.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:05.867+0000 7fbc077fe700 1 --2- 172.21.15.88:0/3075422168 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fbbf405f120 0x7fbbf40615e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:05.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:05.867+0000 7fbc077fe700 1 --2- 172.21.15.88:0/3075422168 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fbbf405f120 0x7fbbf40615e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fbbfc007fd0 tx=0x7fbbfc009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:11:05.879 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:05.871+0000 7fbbeffff700 1 -- 172.21.15.88:0/3075422168 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7fbbf804cdb0 con 0x7fbc08065360 2023-10-18T16:11:06.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:06.071+0000 7fbc0e63a700 1 -- 172.21.15.88:0/3075422168 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fbc080ba3d0 con 0x7fbc08065360 2023-10-18T16:11:06.075 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:06.071+0000 7fbbeffff700 1 -- 172.21.15.88:0/3075422168 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7fbbf80503d0 con 0x7fbc08065360 2023-10-18T16:11:06.075 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:11:06.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:06.073+0000 7fbc0e63a700 1 -- 172.21.15.88:0/3075422168 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fbbf405f120 msgr2=0x7fbbf40615e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:06.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:06.073+0000 7fbc0e63a700 1 --2- 172.21.15.88:0/3075422168 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fbbf405f120 0x7fbbf40615e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fbbfc007fd0 tx=0x7fbbfc009040).stop 2023-10-18T16:11:06.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:06.073+0000 7fbc0e63a700 1 -- 172.21.15.88:0/3075422168 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc08065360 msgr2=0x7fbc080bb860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:06.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:06.073+0000 7fbc0e63a700 1 --2- 172.21.15.88:0/3075422168 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc08065360 0x7fbc080bb860 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7fbbf8009eb0 tx=0x7fbbf8017000).stop 2023-10-18T16:11:06.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:06.073+0000 7fbc0e63a700 1 -- 172.21.15.88:0/3075422168 shutdown_connections 2023-10-18T16:11:06.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:06.073+0000 7fbc0e63a700 1 --2- 172.21.15.88:0/3075422168 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fbbf405f120 0x7fbbf40615e0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:06.079 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:06.073+0000 7fbc0e63a700 1 --2- 172.21.15.88:0/3075422168 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbc0814d8c0 0x7fbc080abf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-10-18T16:11:06.079 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:06.073+0000 7fbc0e63a700 1 --2- 172.21.15.88:0/3075422168 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc08065360 0x7fbc080bb860 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:06.079 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:06.073+0000 7fbc0e63a700 1 --2- 172.21.15.88:0/3075422168 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbc080a9050 0x7fbc080aba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:06.079 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:06.073+0000 7fbc0e63a700 1 -- 172.21.15.88:0/3075422168 >> 172.21.15.88:0/3075422168 conn(0x7fbc0806a810 msgr2=0x7fbc08153ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:11:06.080 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:06.073+0000 7fbc0e63a700 1 -- 172.21.15.88:0/3075422168 shutdown_connections 2023-10-18T16:11:06.080 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:06.073+0000 7fbc0e63a700 1 -- 172.21.15.88:0/3075422168 wait complete. 2023-10-18T16:11:06.088 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:11:08.635 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-18T16:11:08.639 DEBUG:teuthology.orchestra.run.smithi151:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-18T16:11:09.089 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:11:09.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.209+0000 7f2f9a63b700 1 Processor -- start 2023-10-18T16:11:09.213 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.209+0000 7f2f9a63b700 1 -- start start 2023-10-18T16:11:09.215 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.210+0000 7f2f9a63b700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f2f940c6a60 0x7f2f940c6e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:09.215 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.210+0000 7f2f9a63b700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2f94063d70 0x7f2f94153660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:09.216 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.210+0000 7f2f9a63b700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f94151020 con 0x7f2f94153b50 2023-10-18T16:11:09.216 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.210+0000 7f2f9a63b700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f94150090 con 0x7f2f94063d70 2023-10-18T16:11:09.216 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.210+0000 7f2f9a63b700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f94150b90 con 0x7f2f940c6a60 2023-10-18T16:11:09.216 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.210+0000 7f2f98e38700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2f94063d70 0x7f2f94153660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:09.216 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.210+0000 7f2f99e3a700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f2f94153b50 0x7f2f940c2400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:56538/0 (socket says 172.21.15.88:56538) 2023-10-18T16:11:09.216 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.210+0000 7f2f98e38700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2f94063d70 0x7f2f94153660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60894/0 (socket says 172.21.15.88:60894) 2023-10-18T16:11:09.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.210+0000 7f2f99e3a700 1 -- 172.21.15.88:0/3118373791 learned_addr learned my addr 172.21.15.88:0/3118373791 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:11:09.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.210+0000 7f2f99639700 1 --2- 172.21.15.88:0/3118373791 >> v2:172.21.15.151:3300/0 conn(0x7f2f940c6a60 0x7f2f940c6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:09.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.210+0000 7f2f98e38700 1 -- 172.21.15.88:0/3118373791 >> v2:172.21.15.151:3300/0 conn(0x7f2f940c6a60 msgr2=0x7f2f940c6e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:11:09.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.210+0000 7f2f98e38700 1 --2- 172.21.15.88:0/3118373791 >> v2:172.21.15.151:3300/0 conn(0x7f2f940c6a60 0x7f2f940c6e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:09.218 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.210+0000 7f2f98e38700 1 -- 172.21.15.88:0/3118373791 >> v1:172.21.15.88:6789/0 conn(0x7f2f94153b50 legacy=0x7f2f940c2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:09.218 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.210+0000 7f2f98e38700 1 -- 172.21.15.88:0/3118373791 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f940c2f70 con 0x7f2f94063d70 2023-10-18T16:11:09.218 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.210+0000 7f2f93fff700 1 -- 172.21.15.88:0/3118373791 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4037136087 0 0) 0x7f2f84001110 con 0x7f2f94153b50 2023-10-18T16:11:09.218 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.210+0000 7f2f99639700 1 --2- 172.21.15.88:0/3118373791 >> v2:172.21.15.151:3300/0 conn(0x7f2f940c6a60 0x7f2f940c6e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:11:09.218 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.210+0000 7f2f98e38700 1 --2- 172.21.15.88:0/3118373791 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2f94063d70 0x7f2f94153660 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f2f940c43c0 tx=0x7f2f88009060).ready entity=mon.0 client_cookie=87cd7bcfe7c390f8 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:11:09.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.211+0000 7f2f93fff700 1 -- 172.21.15.88:0/3118373791 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f8800b580 con 0x7f2f94063d70 2023-10-18T16:11:09.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.211+0000 7f2f93fff700 1 -- 172.21.15.88:0/3118373791 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f8800b6e0 con 0x7f2f94063d70 2023-10-18T16:11:09.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.211+0000 7f2f93fff700 1 -- 172.21.15.88:0/3118373791 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f8800ba60 con 0x7f2f94063d70 2023-10-18T16:11:09.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.211+0000 7f2f9a63b700 1 -- 172.21.15.88:0/3118373791 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2f94063d70 msgr2=0x7f2f94153660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:09.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.211+0000 7f2f9a63b700 1 --2- 172.21.15.88:0/3118373791 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2f94063d70 0x7f2f94153660 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f2f940c43c0 tx=0x7f2f88009060).stop 2023-10-18T16:11:09.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.211+0000 7f2f9a63b700 1 -- 172.21.15.88:0/3118373791 shutdown_connections 2023-10-18T16:11:09.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.211+0000 7f2f9a63b700 1 --2- 172.21.15.88:0/3118373791 >> v2:172.21.15.151:3300/0 conn(0x7f2f940c6a60 0x7f2f940c6e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:09.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.211+0000 7f2f9a63b700 1 --2- 172.21.15.88:0/3118373791 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2f94063d70 0x7f2f94153660 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:09.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.211+0000 7f2f9a63b700 1 -- 172.21.15.88:0/3118373791 >> 172.21.15.88:0/3118373791 conn(0x7f2f9405e050 msgr2=0x7f2f9405e460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:11:09.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.211+0000 7f2f9a63b700 1 -- 172.21.15.88:0/3118373791 shutdown_connections 2023-10-18T16:11:09.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.211+0000 7f2f9a63b700 1 -- 172.21.15.88:0/3118373791 wait complete. 2023-10-18T16:11:09.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.212+0000 7f2f9a63b700 1 Processor -- start 2023-10-18T16:11:09.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.212+0000 7f2f9a63b700 1 -- start start 2023-10-18T16:11:09.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.213+0000 7f2f9a63b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f94063d70 0x7f2f940ba280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:09.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.213+0000 7f2f9a63b700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2f940c6a60 0x7f2f940c5e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:09.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.213+0000 7f2f9a63b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2f94153b50 0x7f2f94151b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:09.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.213+0000 7f2f9a63b700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f94153120 con 0x7f2f94153b50 2023-10-18T16:11:09.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.213+0000 7f2f9a63b700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f94160820 con 0x7f2f94063d70 2023-10-18T16:11:09.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.213+0000 7f2f9a63b700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f94151610 con 0x7f2f940c6a60 2023-10-18T16:11:09.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.213+0000 7f2f99e3a700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2f94153b50 0x7f2f94151b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:09.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.213+0000 7f2f99e3a700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2f94153b50 0x7f2f94151b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60898/0 (socket says 172.21.15.88:60898) 2023-10-18T16:11:09.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.213+0000 7f2f99e3a700 1 -- 172.21.15.88:0/1683337165 learned_addr learned my addr 172.21.15.88:0/1683337165 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:11:09.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.213+0000 7f2f99639700 1 --2- 172.21.15.88:0/1683337165 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f94063d70 0x7f2f940ba280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:09.226 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.213+0000 7f2f98e38700 1 --2- 172.21.15.88:0/1683337165 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2f940c6a60 0x7f2f940c5e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:09.226 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.213+0000 7f2f99e3a700 1 -- 172.21.15.88:0/1683337165 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2f940c6a60 msgr2=0x7f2f940c5e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:09.226 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.213+0000 7f2f99e3a700 1 --2- 172.21.15.88:0/1683337165 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2f940c6a60 0x7f2f940c5e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:09.226 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.213+0000 7f2f99e3a700 1 -- 172.21.15.88:0/1683337165 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f94063d70 msgr2=0x7f2f940ba280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:09.226 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.213+0000 7f2f99e3a700 1 --2- 172.21.15.88:0/1683337165 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f94063d70 0x7f2f940ba280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:09.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.214+0000 7f2f99e3a700 1 -- 172.21.15.88:0/1683337165 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f880092a0 con 0x7f2f94153b50 2023-10-18T16:11:09.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.214+0000 7f2f99e3a700 1 --2- 172.21.15.88:0/1683337165 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2f94153b50 0x7f2f94151b10 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f2f8400c460 tx=0x7f2f84007e10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:11:09.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.214+0000 7f2f917fa700 1 -- 172.21.15.88:0/1683337165 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f840072e0 con 0x7f2f94153b50 2023-10-18T16:11:09.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.214+0000 7f2f917fa700 1 -- 172.21.15.88:0/1683337165 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f84007440 con 0x7f2f94153b50 2023-10-18T16:11:09.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.214+0000 7f2f9a63b700 1 -- 172.21.15.88:0/1683337165 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f94160ac0 con 0x7f2f94153b50 2023-10-18T16:11:09.228 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.214+0000 7f2f917fa700 1 -- 172.21.15.88:0/1683337165 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f840057a0 con 0x7f2f94153b50 2023-10-18T16:11:09.228 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.214+0000 7f2f9a63b700 1 -- 172.21.15.88:0/1683337165 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f94161080 con 0x7f2f94153b50 2023-10-18T16:11:09.228 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.215+0000 7f2f9a63b700 1 -- 172.21.15.88:0/1683337165 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f94151830 con 0x7f2f94153b50 2023-10-18T16:11:09.228 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.215+0000 7f2f917fa700 1 -- 172.21.15.88:0/1683337165 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f2f84005b50 con 0x7f2f94153b50 2023-10-18T16:11:09.228 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.215+0000 7f2f917fa700 1 --2- 172.21.15.88:0/1683337165 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f2f8005f150 0x7f2f80061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:09.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.215+0000 7f2f917fa700 1 -- 172.21.15.88:0/1683337165 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6725+0+0 (secure 0 0 0) 0x7f2f840799d0 con 0x7f2f94153b50 2023-10-18T16:11:09.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.216+0000 7f2f99639700 1 --2- 172.21.15.88:0/1683337165 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f2f8005f150 0x7f2f80061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:09.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.216+0000 7f2f99639700 1 --2- 172.21.15.88:0/1683337165 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f2f8005f150 0x7f2f80061610 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f2f8c0076a0 tx=0x7f2f8c0098f0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:11:09.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.220+0000 7f2f917fa700 1 -- 172.21.15.88:0/1683337165 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f2f94151830 con 0x7f2f94153b50 2023-10-18T16:11:09.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.421+0000 7f2f9a63b700 1 -- 172.21.15.88:0/1683337165 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2f94151020 con 0x7f2f94153b50 2023-10-18T16:11:09.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.422+0000 7f2f917fa700 1 -- 172.21.15.88:0/1683337165 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7f2f94151020 con 0x7f2f94153b50 2023-10-18T16:11:09.426 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:11:09.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.423+0000 7f2f9a63b700 1 -- 172.21.15.88:0/1683337165 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f2f8005f150 msgr2=0x7f2f80061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:09.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.423+0000 7f2f9a63b700 1 --2- 172.21.15.88:0/1683337165 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f2f8005f150 0x7f2f80061610 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f2f8c0076a0 tx=0x7f2f8c0098f0).stop 2023-10-18T16:11:09.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.423+0000 7f2f9a63b700 1 -- 172.21.15.88:0/1683337165 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2f94153b50 msgr2=0x7f2f94151b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:09.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.423+0000 7f2f9a63b700 1 --2- 172.21.15.88:0/1683337165 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2f94153b50 0x7f2f94151b10 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f2f8400c460 tx=0x7f2f84007e10).stop 2023-10-18T16:11:09.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.423+0000 7f2f9a63b700 1 -- 172.21.15.88:0/1683337165 shutdown_connections 2023-10-18T16:11:09.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.423+0000 7f2f9a63b700 1 --2- 172.21.15.88:0/1683337165 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f2f8005f150 0x7f2f80061610 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:09.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.423+0000 7f2f9a63b700 1 --2- 172.21.15.88:0/1683337165 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2f94153b50 0x7f2f94151b10 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:09.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.423+0000 7f2f9a63b700 1 --2- 172.21.15.88:0/1683337165 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f94063d70 0x7f2f940ba280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:09.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.423+0000 7f2f9a63b700 1 --2- 172.21.15.88:0/1683337165 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2f940c6a60 0x7f2f940c5e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:09.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.423+0000 7f2f9a63b700 1 -- 172.21.15.88:0/1683337165 >> 172.21.15.88:0/1683337165 conn(0x7f2f9405e050 msgr2=0x7f2f940c0f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:11:09.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.424+0000 7f2f9a63b700 1 -- 172.21.15.88:0/1683337165 shutdown_connections 2023-10-18T16:11:09.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:09.424+0000 7f2f9a63b700 1 -- 172.21.15.88:0/1683337165 wait complete. 2023-10-18T16:11:09.439 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:11:10.554 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~95s 2023-10-18T16:11:10.554 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~107s 2023-10-18T16:11:10.554 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~140s 2023-10-18T16:11:10.555 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~121s 2023-10-18T16:11:10.555 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~201s 2023-10-18T16:11:10.555 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.f is failed for ~154s 2023-10-18T16:11:12.440 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:11:12.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.561+0000 7f5aa2d48700 1 Processor -- start 2023-10-18T16:11:12.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.562+0000 7f5aa2d48700 1 -- start start 2023-10-18T16:11:12.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.562+0000 7f5aa2d48700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5a9c0c5660 0x7f5a9c151cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:12.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.562+0000 7f5aa2d48700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f5a9c0bf400 0x7f5a9c1521e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:12.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.562+0000 7f5aa2d48700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a9c067dd0 con 0x7f5a9c151910 2023-10-18T16:11:12.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.562+0000 7f5aa2d48700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a9c067b20 con 0x7f5a9c0c5660 2023-10-18T16:11:12.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.562+0000 7f5aa2d48700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a9c0b9b60 con 0x7f5a9c0bf400 2023-10-18T16:11:12.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.562+0000 7f5a9bfff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5a9c0c5660 0x7f5a9c151cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:12.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.562+0000 7f5aa0ae4700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f5a9c151910 0x7f5a9c068650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:56554/0 (socket says 172.21.15.88:56554) 2023-10-18T16:11:12.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.563+0000 7f5aa0ae4700 1 -- 172.21.15.88:0/388799012 learned_addr learned my addr 172.21.15.88:0/388799012 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:11:12.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.563+0000 7f5aa12e5700 1 --2- 172.21.15.88:0/388799012 >> v2:172.21.15.151:3300/0 conn(0x7f5a9c0bf400 0x7f5a9c1521e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:12.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.563+0000 7f5a9bfff700 1 -- 172.21.15.88:0/388799012 >> v2:172.21.15.151:3300/0 conn(0x7f5a9c0bf400 msgr2=0x7f5a9c1521e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:11:12.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.563+0000 7f5a9bfff700 1 --2- 172.21.15.88:0/388799012 >> v2:172.21.15.151:3300/0 conn(0x7f5a9c0bf400 0x7f5a9c1521e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:12.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.563+0000 7f5a9bfff700 1 -- 172.21.15.88:0/388799012 >> v1:172.21.15.88:6789/0 conn(0x7f5a9c151910 legacy=0x7f5a9c068650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:12.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.563+0000 7f5a9b7fe700 1 -- 172.21.15.88:0/388799012 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4089511221 0 0) 0x7f5a9c067dd0 con 0x7f5a9c151910 2023-10-18T16:11:12.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.563+0000 7f5a9bfff700 1 -- 172.21.15.88:0/388799012 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a9c0b9ff0 con 0x7f5a9c0c5660 2023-10-18T16:11:12.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.563+0000 7f5a9bfff700 1 --2- 172.21.15.88:0/388799012 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5a9c0c5660 0x7f5a9c151cf0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f5a9c06ac10 tx=0x7f5a90009060).ready entity=mon.0 client_cookie=bbc7de578e8fd999 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:11:12.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.563+0000 7f5a9b7fe700 1 -- 172.21.15.88:0/388799012 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a9000b610 con 0x7f5a9c0c5660 2023-10-18T16:11:12.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.563+0000 7f5a9b7fe700 1 -- 172.21.15.88:0/388799012 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a9000b770 con 0x7f5a9c0c5660 2023-10-18T16:11:12.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.563+0000 7f5a9b7fe700 1 -- 172.21.15.88:0/388799012 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a9000baf0 con 0x7f5a9c0c5660 2023-10-18T16:11:12.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.563+0000 7f5aa2d48700 1 -- 172.21.15.88:0/388799012 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5a9c0c5660 msgr2=0x7f5a9c151cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:12.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.563+0000 7f5aa2d48700 1 --2- 172.21.15.88:0/388799012 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5a9c0c5660 0x7f5a9c151cf0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f5a9c06ac10 tx=0x7f5a90009060).stop 2023-10-18T16:11:12.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.564+0000 7f5aa2d48700 1 -- 172.21.15.88:0/388799012 shutdown_connections 2023-10-18T16:11:12.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.564+0000 7f5aa2d48700 1 --2- 172.21.15.88:0/388799012 >> v2:172.21.15.151:3300/0 conn(0x7f5a9c0bf400 0x7f5a9c1521e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:12.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.564+0000 7f5aa2d48700 1 --2- 172.21.15.88:0/388799012 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5a9c0c5660 0x7f5a9c151cf0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:12.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.564+0000 7f5aa2d48700 1 -- 172.21.15.88:0/388799012 >> 172.21.15.88:0/388799012 conn(0x7f5a9c0c4940 msgr2=0x7f5a9c0c4d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:11:12.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.564+0000 7f5aa2d48700 1 -- 172.21.15.88:0/388799012 shutdown_connections 2023-10-18T16:11:12.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.564+0000 7f5aa2d48700 1 -- 172.21.15.88:0/388799012 wait complete. 2023-10-18T16:11:12.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.565+0000 7f5aa2d48700 1 Processor -- start 2023-10-18T16:11:12.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.565+0000 7f5aa2d48700 1 -- start start 2023-10-18T16:11:12.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.565+0000 7f5aa2d48700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5a9c0bf400 0x7f5a9c0687f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:12.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.566+0000 7f5aa2d48700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5a9c0c5660 0x7f5a9c161350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:12.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.566+0000 7f5aa2d48700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5a9c151910 0x7f5a9c161840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:12.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.566+0000 7f5aa2d48700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a9c0bdb70 con 0x7f5a9c0bf400 2023-10-18T16:11:12.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.566+0000 7f5aa2d48700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a9c161d30 con 0x7f5a9c0c5660 2023-10-18T16:11:12.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.566+0000 7f5aa2d48700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a9c0be340 con 0x7f5a9c151910 2023-10-18T16:11:12.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.566+0000 7f5aa0ae4700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5a9c0bf400 0x7f5a9c0687f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:12.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.566+0000 7f5aa0ae4700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5a9c0bf400 0x7f5a9c0687f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60916/0 (socket says 172.21.15.88:60916) 2023-10-18T16:11:12.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.566+0000 7f5aa0ae4700 1 -- 172.21.15.88:0/1950991906 learned_addr learned my addr 172.21.15.88:0/1950991906 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:11:12.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.566+0000 7f5a9bfff700 1 --2- 172.21.15.88:0/1950991906 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5a9c0c5660 0x7f5a9c161350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:12.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.566+0000 7f5aa0ae4700 1 -- 172.21.15.88:0/1950991906 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5a9c151910 msgr2=0x7f5a9c161840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:12.579 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.566+0000 7f5aa12e5700 1 --2- 172.21.15.88:0/1950991906 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5a9c151910 0x7f5a9c161840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:12.579 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.566+0000 7f5aa0ae4700 1 --2- 172.21.15.88:0/1950991906 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5a9c151910 0x7f5a9c161840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:12.579 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.566+0000 7f5aa0ae4700 1 -- 172.21.15.88:0/1950991906 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5a9c0c5660 msgr2=0x7f5a9c161350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:12.579 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.566+0000 7f5aa0ae4700 1 --2- 172.21.15.88:0/1950991906 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5a9c0c5660 0x7f5a9c161350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:12.579 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.566+0000 7f5aa0ae4700 1 -- 172.21.15.88:0/1950991906 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a900092a0 con 0x7f5a9c0bf400 2023-10-18T16:11:12.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.566+0000 7f5a9bfff700 1 --2- 172.21.15.88:0/1950991906 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5a9c0c5660 0x7f5a9c161350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-10-18T16:11:12.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.566+0000 7f5aa0ae4700 1 --2- 172.21.15.88:0/1950991906 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5a9c0bf400 0x7f5a9c0687f0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f5a8c00c810 tx=0x7f5a8c001570).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:11:12.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.566+0000 7f5a98ff9700 1 -- 172.21.15.88:0/1950991906 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a8c005d30 con 0x7f5a9c0bf400 2023-10-18T16:11:12.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.566+0000 7f5a98ff9700 1 -- 172.21.15.88:0/1950991906 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a8c005e90 con 0x7f5a9c0bf400 2023-10-18T16:11:12.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.566+0000 7f5aa2d48700 1 -- 172.21.15.88:0/1950991906 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a9c0be620 con 0x7f5a9c0bf400 2023-10-18T16:11:12.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.566+0000 7f5a98ff9700 1 -- 172.21.15.88:0/1950991906 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a8c007320 con 0x7f5a9c0bf400 2023-10-18T16:11:12.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.566+0000 7f5aa2d48700 1 -- 172.21.15.88:0/1950991906 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a9c0bd800 con 0x7f5a9c0bf400 2023-10-18T16:11:12.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.567+0000 7f5aa2d48700 1 -- 172.21.15.88:0/1950991906 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a9c0bde50 con 0x7f5a9c0bf400 2023-10-18T16:11:12.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.568+0000 7f5a98ff9700 1 -- 172.21.15.88:0/1950991906 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f5a8c007710 con 0x7f5a9c0bf400 2023-10-18T16:11:12.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.568+0000 7f5a98ff9700 1 --2- 172.21.15.88:0/1950991906 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f5a8805f150 0x7f5a88061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:12.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.568+0000 7f5a98ff9700 1 -- 172.21.15.88:0/1950991906 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6725+0+0 (secure 0 0 0) 0x7f5a8c07a520 con 0x7f5a9c0bf400 2023-10-18T16:11:12.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.568+0000 7f5a9bfff700 1 --2- 172.21.15.88:0/1950991906 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f5a8805f150 0x7f5a88061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:12.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.569+0000 7f5a9bfff700 1 --2- 172.21.15.88:0/1950991906 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f5a8805f150 0x7f5a88061610 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f5a9c143de0 tx=0x7f5a90017000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:11:12.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.574+0000 7f5a98ff9700 1 -- 172.21.15.88:0/1950991906 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f5a8c04b590 con 0x7f5a9c0bf400 2023-10-18T16:11:12.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.774+0000 7f5aa2d48700 1 -- 172.21.15.88:0/1950991906 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f5a9c0bd360 con 0x7f5a9c0bf400 2023-10-18T16:11:12.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.774+0000 7f5a98ff9700 1 -- 172.21.15.88:0/1950991906 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7f5a8c0016b0 con 0x7f5a9c0bf400 2023-10-18T16:11:12.778 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:11:12.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.776+0000 7f5aa2d48700 1 -- 172.21.15.88:0/1950991906 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f5a8805f150 msgr2=0x7f5a88061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:12.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.776+0000 7f5aa2d48700 1 --2- 172.21.15.88:0/1950991906 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f5a8805f150 0x7f5a88061610 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f5a9c143de0 tx=0x7f5a90017000).stop 2023-10-18T16:11:12.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.776+0000 7f5aa2d48700 1 -- 172.21.15.88:0/1950991906 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5a9c0bf400 msgr2=0x7f5a9c0687f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:12.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.776+0000 7f5aa2d48700 1 --2- 172.21.15.88:0/1950991906 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5a9c0bf400 0x7f5a9c0687f0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f5a8c00c810 tx=0x7f5a8c001570).stop 2023-10-18T16:11:12.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.776+0000 7f5aa2d48700 1 -- 172.21.15.88:0/1950991906 shutdown_connections 2023-10-18T16:11:12.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.776+0000 7f5aa2d48700 1 --2- 172.21.15.88:0/1950991906 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f5a8805f150 0x7f5a88061610 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:12.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.776+0000 7f5aa2d48700 1 --2- 172.21.15.88:0/1950991906 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5a9c151910 0x7f5a9c161840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:12.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.776+0000 7f5aa2d48700 1 --2- 172.21.15.88:0/1950991906 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5a9c0bf400 0x7f5a9c0687f0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:12.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.776+0000 7f5aa2d48700 1 --2- 172.21.15.88:0/1950991906 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5a9c0c5660 0x7f5a9c161350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:12.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.776+0000 7f5aa2d48700 1 -- 172.21.15.88:0/1950991906 >> 172.21.15.88:0/1950991906 conn(0x7f5a9c0c4940 msgr2=0x7f5a9c0693d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:11:12.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.776+0000 7f5aa2d48700 1 -- 172.21.15.88:0/1950991906 shutdown_connections 2023-10-18T16:11:12.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:12.776+0000 7f5aa2d48700 1 -- 172.21.15.88:0/1950991906 wait complete. 2023-10-18T16:11:12.791 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:11:15.792 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:11:15.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:15.914+0000 7f3f255e6700 1 Processor -- start 2023-10-18T16:11:15.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:15.914+0000 7f3f255e6700 1 -- start start 2023-10-18T16:11:15.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:15.915+0000 7f3f255e6700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3f2014d8c0 0x7f3f20064440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:15.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:15.915+0000 7f3f255e6700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3f200a9050 0x7f3f20064930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:15.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:15.915+0000 7f3f255e6700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f20064150 con 0x7f3f20065360 2023-10-18T16:11:15.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:15.915+0000 7f3f255e6700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f20153a40 con 0x7f3f2014d8c0 2023-10-18T16:11:15.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:15.915+0000 7f3f255e6700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f20063e70 con 0x7f3f200a9050 2023-10-18T16:11:15.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:15.915+0000 7f3f1effd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3f2014d8c0 0x7f3f20064440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:15.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:15.915+0000 7f3f1effd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3f2014d8c0 0x7f3f20064440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60918/0 (socket says 172.21.15.88:60918) 2023-10-18T16:11:15.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:15.915+0000 7f3f1effd700 1 -- 172.21.15.88:0/221081899 learned_addr learned my addr 172.21.15.88:0/221081899 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:11:15.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:15.915+0000 7f3f1f7fe700 1 --1- 172.21.15.88:0/221081899 >> v1:172.21.15.88:6789/0 conn(0x7f3f20065360 0x7f3f20153ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:56558/0 (socket says 172.21.15.88:56558) 2023-10-18T16:11:15.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:15.915+0000 7f3f1effd700 1 -- 172.21.15.88:0/221081899 >> v2:172.21.15.151:3300/0 conn(0x7f3f200a9050 msgr2=0x7f3f20064930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:11:15.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:15.915+0000 7f3f167fc700 1 --2- 172.21.15.88:0/221081899 >> v2:172.21.15.151:3300/0 conn(0x7f3f200a9050 0x7f3f20064930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:15.923 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:15.915+0000 7f3f1e7fc700 1 -- 172.21.15.88:0/221081899 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1551125487 0 0) 0x7f3f20064150 con 0x7f3f20065360 2023-10-18T16:11:15.923 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:15.915+0000 7f3f1effd700 1 --2- 172.21.15.88:0/221081899 >> v2:172.21.15.151:3300/0 conn(0x7f3f200a9050 0x7f3f20064930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:15.923 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:15.915+0000 7f3f1effd700 1 -- 172.21.15.88:0/221081899 >> v1:172.21.15.88:6789/0 conn(0x7f3f20065360 legacy=0x7f3f20153ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:15.923 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:15.915+0000 7f3f1effd700 1 -- 172.21.15.88:0/221081899 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f200c72c0 con 0x7f3f2014d8c0 2023-10-18T16:11:15.923 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:15.915+0000 7f3f167fc700 1 --2- 172.21.15.88:0/221081899 >> v2:172.21.15.151:3300/0 conn(0x7f3f200a9050 0x7f3f20064930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-10-18T16:11:15.924 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:15.915+0000 7f3f1effd700 1 --2- 172.21.15.88:0/221081899 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f2014d8c0 0x7f3f20064440 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f3f200637d0 tx=0x7f3f100090f0).ready entity=mon.0 client_cookie=adff725b035e5ef9 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:11:15.924 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:15.915+0000 7f3f1e7fc700 1 -- 172.21.15.88:0/221081899 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f1000b600 con 0x7f3f2014d8c0 2023-10-18T16:11:15.924 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:15.915+0000 7f3f1e7fc700 1 -- 172.21.15.88:0/221081899 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f1000b760 con 0x7f3f2014d8c0 2023-10-18T16:11:15.924 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:15.915+0000 7f3f1e7fc700 1 -- 172.21.15.88:0/221081899 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f1000bb00 con 0x7f3f2014d8c0 2023-10-18T16:11:15.924 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:15.916+0000 7f3f255e6700 1 -- 172.21.15.88:0/221081899 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f2014d8c0 msgr2=0x7f3f20064440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:15.924 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:15.916+0000 7f3f255e6700 1 --2- 172.21.15.88:0/221081899 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f2014d8c0 0x7f3f20064440 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f3f200637d0 tx=0x7f3f100090f0).stop 2023-10-18T16:11:15.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:15.916+0000 7f3f255e6700 1 -- 172.21.15.88:0/221081899 shutdown_connections 2023-10-18T16:11:15.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:15.916+0000 7f3f255e6700 1 --2- 172.21.15.88:0/221081899 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f2014d8c0 0x7f3f20064440 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:15.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:15.916+0000 7f3f255e6700 1 --2- 172.21.15.88:0/221081899 >> v2:172.21.15.151:3300/0 conn(0x7f3f200a9050 0x7f3f20064930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:15.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:15.916+0000 7f3f255e6700 1 -- 172.21.15.88:0/221081899 >> 172.21.15.88:0/221081899 conn(0x7f3f2006a810 msgr2=0x7f3f200c6a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:11:15.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:15.916+0000 7f3f255e6700 1 -- 172.21.15.88:0/221081899 shutdown_connections 2023-10-18T16:11:15.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:15.916+0000 7f3f255e6700 1 -- 172.21.15.88:0/221081899 wait complete. 2023-10-18T16:11:15.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:15.917+0000 7f3f255e6700 1 Processor -- start 2023-10-18T16:11:15.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:15.917+0000 7f3f255e6700 1 -- start start 2023-10-18T16:11:15.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:15.918+0000 7f3f255e6700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3f20065360 0x7f3f2014eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:15.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:15.918+0000 7f3f255e6700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3f200a9050 0x7f3f201608e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:15.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:15.918+0000 7f3f255e6700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f2014d8c0 0x7f3f200abd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:15.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:15.918+0000 7f3f255e6700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f2014eff0 con 0x7f3f2014d8c0 2023-10-18T16:11:15.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:15.918+0000 7f3f255e6700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f20160140 con 0x7f3f20065360 2023-10-18T16:11:15.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:15.918+0000 7f3f255e6700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f201504e0 con 0x7f3f200a9050 2023-10-18T16:11:15.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:15.918+0000 7f3f1f7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f2014d8c0 0x7f3f200abd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:15.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:15.918+0000 7f3f1f7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f2014d8c0 0x7f3f200abd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60932/0 (socket says 172.21.15.88:60932) 2023-10-18T16:11:15.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:15.918+0000 7f3f1f7fe700 1 -- 172.21.15.88:0/458973632 learned_addr learned my addr 172.21.15.88:0/458973632 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:11:15.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:15.918+0000 7f3f1effd700 1 --2- 172.21.15.88:0/458973632 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3f20065360 0x7f3f2014eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:15.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:15.918+0000 7f3f167fc700 1 --2- 172.21.15.88:0/458973632 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3f200a9050 0x7f3f201608e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:15.931 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:15.918+0000 7f3f1f7fe700 1 -- 172.21.15.88:0/458973632 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3f200a9050 msgr2=0x7f3f201608e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:15.931 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:15.918+0000 7f3f1f7fe700 1 --2- 172.21.15.88:0/458973632 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3f200a9050 0x7f3f201608e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:15.931 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:15.918+0000 7f3f1f7fe700 1 -- 172.21.15.88:0/458973632 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3f20065360 msgr2=0x7f3f2014eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:15.931 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:15.918+0000 7f3f1f7fe700 1 --2- 172.21.15.88:0/458973632 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3f20065360 0x7f3f2014eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:15.931 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:15.918+0000 7f3f1f7fe700 1 -- 172.21.15.88:0/458973632 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f10009370 con 0x7f3f2014d8c0 2023-10-18T16:11:15.932 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:15.919+0000 7f3f1f7fe700 1 --2- 172.21.15.88:0/458973632 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f2014d8c0 0x7f3f200abd20 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f3f1800c810 tx=0x7f3f18007f10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:11:15.932 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:15.919+0000 7f3f177fe700 1 -- 172.21.15.88:0/458973632 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f18005c70 con 0x7f3f2014d8c0 2023-10-18T16:11:15.932 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:15.919+0000 7f3f177fe700 1 -- 172.21.15.88:0/458973632 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f18005df0 con 0x7f3f2014d8c0 2023-10-18T16:11:15.932 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:15.919+0000 7f3f177fe700 1 -- 172.21.15.88:0/458973632 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f18007360 con 0x7f3f2014d8c0 2023-10-18T16:11:15.932 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:15.919+0000 7f3f255e6700 1 -- 172.21.15.88:0/458973632 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f201507a0 con 0x7f3f2014d8c0 2023-10-18T16:11:15.933 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:15.919+0000 7f3f255e6700 1 -- 172.21.15.88:0/458973632 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f20150d60 con 0x7f3f2014d8c0 2023-10-18T16:11:15.933 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:15.920+0000 7f3f255e6700 1 -- 172.21.15.88:0/458973632 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f2006c4e0 con 0x7f3f2014d8c0 2023-10-18T16:11:15.933 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:15.920+0000 7f3f177fe700 1 -- 172.21.15.88:0/458973632 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f3f18007750 con 0x7f3f2014d8c0 2023-10-18T16:11:15.933 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:15.920+0000 7f3f177fe700 1 --2- 172.21.15.88:0/458973632 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f3f0005f150 0x7f3f00061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:15.933 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:15.920+0000 7f3f177fe700 1 -- 172.21.15.88:0/458973632 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6725+0+0 (secure 0 0 0) 0x7f3f1807a500 con 0x7f3f2014d8c0 2023-10-18T16:11:15.933 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:15.921+0000 7f3f1effd700 1 --2- 172.21.15.88:0/458973632 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f3f0005f150 0x7f3f00061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:15.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:15.921+0000 7f3f1effd700 1 --2- 172.21.15.88:0/458973632 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f3f0005f150 0x7f3f00061610 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f3f1000b3c0 tx=0x7f3f10017000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:11:15.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:15.925+0000 7f3f177fe700 1 -- 172.21.15.88:0/458973632 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f3f1804b570 con 0x7f3f2014d8c0 2023-10-18T16:11:16.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:16.126+0000 7f3f255e6700 1 -- 172.21.15.88:0/458973632 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3f20065140 con 0x7f3f2014d8c0 2023-10-18T16:11:16.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:16.126+0000 7f3f177fe700 1 -- 172.21.15.88:0/458973632 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7f3f1804eb90 con 0x7f3f2014d8c0 2023-10-18T16:11:16.131 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:11:16.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:16.128+0000 7f3f255e6700 1 -- 172.21.15.88:0/458973632 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f3f0005f150 msgr2=0x7f3f00061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:16.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:16.128+0000 7f3f255e6700 1 --2- 172.21.15.88:0/458973632 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f3f0005f150 0x7f3f00061610 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f3f1000b3c0 tx=0x7f3f10017000).stop 2023-10-18T16:11:16.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:16.128+0000 7f3f255e6700 1 -- 172.21.15.88:0/458973632 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f2014d8c0 msgr2=0x7f3f200abd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:16.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:16.128+0000 7f3f255e6700 1 --2- 172.21.15.88:0/458973632 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f2014d8c0 0x7f3f200abd20 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f3f1800c810 tx=0x7f3f18007f10).stop 2023-10-18T16:11:16.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:16.128+0000 7f3f255e6700 1 -- 172.21.15.88:0/458973632 shutdown_connections 2023-10-18T16:11:16.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:16.128+0000 7f3f255e6700 1 --2- 172.21.15.88:0/458973632 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f3f0005f150 0x7f3f00061610 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:16.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:16.128+0000 7f3f255e6700 1 --2- 172.21.15.88:0/458973632 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f2014d8c0 0x7f3f200abd20 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:16.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:16.128+0000 7f3f255e6700 1 --2- 172.21.15.88:0/458973632 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3f20065360 0x7f3f2014eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:16.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:16.128+0000 7f3f255e6700 1 --2- 172.21.15.88:0/458973632 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3f200a9050 0x7f3f201608e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:16.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:16.128+0000 7f3f255e6700 1 -- 172.21.15.88:0/458973632 >> 172.21.15.88:0/458973632 conn(0x7f3f2006a810 msgr2=0x7f3f200c4cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:11:16.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:16.128+0000 7f3f255e6700 1 -- 172.21.15.88:0/458973632 shutdown_connections 2023-10-18T16:11:16.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:16.128+0000 7f3f255e6700 1 -- 172.21.15.88:0/458973632 wait complete. 2023-10-18T16:11:16.143 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:11:16.866 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~101s 2023-10-18T16:11:16.866 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~114s 2023-10-18T16:11:16.866 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~147s 2023-10-18T16:11:16.867 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~127s 2023-10-18T16:11:16.867 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~208s 2023-10-18T16:11:16.867 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.f is failed for ~160s 2023-10-18T16:11:19.144 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:11:19.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.266+0000 7f9e9e704700 1 Processor -- start 2023-10-18T16:11:19.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.266+0000 7f9e9e704700 1 -- start start 2023-10-18T16:11:19.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.266+0000 7f9e9e704700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9e980bf200 0x7f9e980bf5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:19.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.266+0000 7f9e9e704700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f9e980bcfb0 0x7f9e981511f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:19.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.266+0000 7f9e9e704700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9e980bfb90 0x7f9e980691a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:19.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.266+0000 7f9e9e704700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e981516e0 con 0x7f9e980bfb90 2023-10-18T16:11:19.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.266+0000 7f9e9e704700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e980be390 con 0x7f9e980bcfb0 2023-10-18T16:11:19.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.266+0000 7f9e9e704700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e980be820 con 0x7f9e980bf200 2023-10-18T16:11:19.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.267+0000 7f9e9cca1700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9e980bfb90 0x7f9e980691a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:19.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.267+0000 7f9e9cca1700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9e980bfb90 0x7f9e980691a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59500/0 (socket says 172.21.15.88:59500) 2023-10-18T16:11:19.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.267+0000 7f9e9cca1700 1 -- 172.21.15.88:0/173083982 learned_addr learned my addr 172.21.15.88:0/173083982 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:11:19.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.267+0000 7f9e97fff700 1 --2- 172.21.15.88:0/173083982 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9e980bf200 0x7f9e980bf5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:19.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.267+0000 7f9e977fe700 1 --2- 172.21.15.88:0/173083982 >> v2:172.21.15.151:3300/0 conn(0x7f9e980bcfb0 0x7f9e981511f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:19.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.267+0000 7f9e9cca1700 1 -- 172.21.15.88:0/173083982 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9e980bf200 msgr2=0x7f9e980bf5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:11:19.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.267+0000 7f9e9cca1700 1 --2- 172.21.15.88:0/173083982 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9e980bf200 0x7f9e980bf5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:19.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.267+0000 7f9e9cca1700 1 -- 172.21.15.88:0/173083982 >> v2:172.21.15.151:3300/0 conn(0x7f9e980bcfb0 msgr2=0x7f9e981511f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:11:19.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.267+0000 7f9e9cca1700 1 --2- 172.21.15.88:0/173083982 >> v2:172.21.15.151:3300/0 conn(0x7f9e980bcfb0 0x7f9e981511f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:19.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.267+0000 7f9e9cca1700 1 -- 172.21.15.88:0/173083982 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e980bd4a0 con 0x7f9e980bfb90 2023-10-18T16:11:19.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.267+0000 7f9e9cca1700 1 --2- 172.21.15.88:0/173083982 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e980bfb90 0x7f9e980691a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f9e98068520 tx=0x7f9e90009840).ready entity=mon.0 client_cookie=924eace40d390786 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:11:19.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.267+0000 7f9e96ffd700 1 -- 172.21.15.88:0/173083982 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e9000e070 con 0x7f9e980bfb90 2023-10-18T16:11:19.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.267+0000 7f9e96ffd700 1 -- 172.21.15.88:0/173083982 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e9000ba00 con 0x7f9e980bfb90 2023-10-18T16:11:19.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.267+0000 7f9e96ffd700 1 -- 172.21.15.88:0/173083982 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e9000bd80 con 0x7f9e980bfb90 2023-10-18T16:11:19.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.268+0000 7f9e9e704700 1 -- 172.21.15.88:0/173083982 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e980bfb90 msgr2=0x7f9e980691a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:19.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.268+0000 7f9e9e704700 1 --2- 172.21.15.88:0/173083982 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e980bfb90 0x7f9e980691a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f9e98068520 tx=0x7f9e90009840).stop 2023-10-18T16:11:19.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.268+0000 7f9e9e704700 1 -- 172.21.15.88:0/173083982 shutdown_connections 2023-10-18T16:11:19.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.268+0000 7f9e9e704700 1 --2- 172.21.15.88:0/173083982 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e980bfb90 0x7f9e980691a0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:19.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.268+0000 7f9e9e704700 1 --2- 172.21.15.88:0/173083982 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9e980bf200 0x7f9e980bf5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:19.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.268+0000 7f9e9e704700 1 --2- 172.21.15.88:0/173083982 >> v2:172.21.15.151:3300/0 conn(0x7f9e980bcfb0 0x7f9e981511f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:19.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.268+0000 7f9e9e704700 1 -- 172.21.15.88:0/173083982 >> 172.21.15.88:0/173083982 conn(0x7f9e98062650 msgr2=0x7f9e98062a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:11:19.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.268+0000 7f9e9e704700 1 -- 172.21.15.88:0/173083982 shutdown_connections 2023-10-18T16:11:19.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.268+0000 7f9e9e704700 1 -- 172.21.15.88:0/173083982 wait complete. 2023-10-18T16:11:19.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.269+0000 7f9e9e704700 1 Processor -- start 2023-10-18T16:11:19.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.269+0000 7f9e9e704700 1 -- start start 2023-10-18T16:11:19.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.270+0000 7f9e9e704700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9e980bcfb0 0x7f9e9814b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:19.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.270+0000 7f9e9e704700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e980bf200 0x7f9e980bbed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:19.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.270+0000 7f9e9e704700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9e980bfb90 0x7f9e98147730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:19.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.270+0000 7f9e9e704700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e980b6420 con 0x7f9e980bf200 2023-10-18T16:11:19.281 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.270+0000 7f9e9e704700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e9814d0e0 con 0x7f9e980bfb90 2023-10-18T16:11:19.281 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.270+0000 7f9e9e704700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e9814deb0 con 0x7f9e980bcfb0 2023-10-18T16:11:19.281 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.270+0000 7f9e977fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e980bf200 0x7f9e980bbed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:19.281 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.270+0000 7f9e977fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e980bf200 0x7f9e980bbed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59510/0 (socket says 172.21.15.88:59510) 2023-10-18T16:11:19.281 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.270+0000 7f9e977fe700 1 -- 172.21.15.88:0/874959617 learned_addr learned my addr 172.21.15.88:0/874959617 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:11:19.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.270+0000 7f9e9cca1700 1 --2- 172.21.15.88:0/874959617 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9e980bfb90 0x7f9e98147730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:19.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.270+0000 7f9e97fff700 1 --2- 172.21.15.88:0/874959617 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9e980bcfb0 0x7f9e9814b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:19.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.270+0000 7f9e977fe700 1 -- 172.21.15.88:0/874959617 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9e980bcfb0 msgr2=0x7f9e9814b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:19.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.270+0000 7f9e977fe700 1 --2- 172.21.15.88:0/874959617 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9e980bcfb0 0x7f9e9814b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:19.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.270+0000 7f9e977fe700 1 -- 172.21.15.88:0/874959617 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9e980bfb90 msgr2=0x7f9e98147730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:19.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.270+0000 7f9e977fe700 1 --2- 172.21.15.88:0/874959617 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9e980bfb90 0x7f9e98147730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:19.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.270+0000 7f9e977fe700 1 -- 172.21.15.88:0/874959617 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e90009ae0 con 0x7f9e980bf200 2023-10-18T16:11:19.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.270+0000 7f9e977fe700 1 --2- 172.21.15.88:0/874959617 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e980bf200 0x7f9e980bbed0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f9e8c00e900 tx=0x7f9e8c009060).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:11:19.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.271+0000 7f9e7ffff700 1 -- 172.21.15.88:0/874959617 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e8c00c620 con 0x7f9e980bf200 2023-10-18T16:11:19.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.271+0000 7f9e7ffff700 1 -- 172.21.15.88:0/874959617 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e8c00c780 con 0x7f9e980bf200 2023-10-18T16:11:19.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.271+0000 7f9e7ffff700 1 -- 172.21.15.88:0/874959617 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e8c00cb00 con 0x7f9e980bf200 2023-10-18T16:11:19.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.271+0000 7f9e9e704700 1 -- 172.21.15.88:0/874959617 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e98063430 con 0x7f9e980bf200 2023-10-18T16:11:19.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.271+0000 7f9e9e704700 1 -- 172.21.15.88:0/874959617 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e9814d610 con 0x7f9e980bf200 2023-10-18T16:11:19.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.272+0000 7f9e9e704700 1 -- 172.21.15.88:0/874959617 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e9806c4e0 con 0x7f9e980bf200 2023-10-18T16:11:19.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.272+0000 7f9e7ffff700 1 -- 172.21.15.88:0/874959617 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f9e8c01b400 con 0x7f9e980bf200 2023-10-18T16:11:19.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.272+0000 7f9e7ffff700 1 --2- 172.21.15.88:0/874959617 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f9e8405f180 0x7f9e84061640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:19.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.272+0000 7f9e7ffff700 1 -- 172.21.15.88:0/874959617 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6725+0+0 (secure 0 0 0) 0x7f9e8c07a870 con 0x7f9e980bf200 2023-10-18T16:11:19.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.273+0000 7f9e97fff700 1 --2- 172.21.15.88:0/874959617 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f9e8405f180 0x7f9e84061640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:19.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.273+0000 7f9e97fff700 1 --2- 172.21.15.88:0/874959617 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f9e8405f180 0x7f9e84061640 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f9e88009c10 tx=0x7f9e88009060).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:11:19.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.277+0000 7f9e7ffff700 1 -- 172.21.15.88:0/874959617 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f9e8c04b880 con 0x7f9e980bf200 2023-10-18T16:11:19.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.476+0000 7f9e9e704700 1 -- 172.21.15.88:0/874959617 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f9e980649e0 con 0x7f9e980bf200 2023-10-18T16:11:19.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.476+0000 7f9e7ffff700 1 -- 172.21.15.88:0/874959617 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7f9e8c04eea0 con 0x7f9e980bf200 2023-10-18T16:11:19.480 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:11:19.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.479+0000 7f9e9e704700 1 -- 172.21.15.88:0/874959617 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f9e8405f180 msgr2=0x7f9e84061640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:19.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.479+0000 7f9e9e704700 1 --2- 172.21.15.88:0/874959617 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f9e8405f180 0x7f9e84061640 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f9e88009c10 tx=0x7f9e88009060).stop 2023-10-18T16:11:19.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.479+0000 7f9e9e704700 1 -- 172.21.15.88:0/874959617 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e980bf200 msgr2=0x7f9e980bbed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:19.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.479+0000 7f9e9e704700 1 --2- 172.21.15.88:0/874959617 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e980bf200 0x7f9e980bbed0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f9e8c00e900 tx=0x7f9e8c009060).stop 2023-10-18T16:11:19.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.479+0000 7f9e9e704700 1 -- 172.21.15.88:0/874959617 shutdown_connections 2023-10-18T16:11:19.484 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.479+0000 7f9e9e704700 1 --2- 172.21.15.88:0/874959617 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f9e8405f180 0x7f9e84061640 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:19.484 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.479+0000 7f9e9e704700 1 --2- 172.21.15.88:0/874959617 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9e980bfb90 0x7f9e98147730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:19.484 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.479+0000 7f9e9e704700 1 --2- 172.21.15.88:0/874959617 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9e980bcfb0 0x7f9e9814b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:19.484 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.479+0000 7f9e9e704700 1 --2- 172.21.15.88:0/874959617 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e980bf200 0x7f9e980bbed0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:19.484 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.479+0000 7f9e9e704700 1 -- 172.21.15.88:0/874959617 >> 172.21.15.88:0/874959617 conn(0x7f9e98062650 msgr2=0x7f9e981518e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:11:19.484 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.479+0000 7f9e9e704700 1 -- 172.21.15.88:0/874959617 shutdown_connections 2023-10-18T16:11:19.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:19.479+0000 7f9e9e704700 1 -- 172.21.15.88:0/874959617 wait complete. 2023-10-18T16:11:19.493 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:11:22.495 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:11:22.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.612+0000 7fd89c285700 1 Processor -- start 2023-10-18T16:11:22.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.612+0000 7fd89c285700 1 -- start start 2023-10-18T16:11:22.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.613+0000 7fd89c285700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd8940c5660 0x7fd894151cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:22.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.613+0000 7fd89c285700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fd8940bf400 0x7fd8941521e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:22.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.613+0000 7fd89c285700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd894067dd0 con 0x7fd894151910 2023-10-18T16:11:22.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.613+0000 7fd89c285700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd894067b20 con 0x7fd8940c5660 2023-10-18T16:11:22.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.613+0000 7fd89c285700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8940b9b60 con 0x7fd8940bf400 2023-10-18T16:11:22.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.613+0000 7fd899820700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd8940c5660 0x7fd894151cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:22.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.613+0000 7fd899820700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd8940c5660 0x7fd894151cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59518/0 (socket says 172.21.15.88:59518) 2023-10-18T16:11:22.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.613+0000 7fd899820700 1 -- 172.21.15.88:0/1756123281 learned_addr learned my addr 172.21.15.88:0/1756123281 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:11:22.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.613+0000 7fd89a822700 1 --2- 172.21.15.88:0/1756123281 >> v2:172.21.15.151:3300/0 conn(0x7fd8940bf400 0x7fd8941521e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:22.620 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.613+0000 7fd899820700 1 -- 172.21.15.88:0/1756123281 >> v2:172.21.15.151:3300/0 conn(0x7fd8940bf400 msgr2=0x7fd8941521e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:11:22.620 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.613+0000 7fd899820700 1 --2- 172.21.15.88:0/1756123281 >> v2:172.21.15.151:3300/0 conn(0x7fd8940bf400 0x7fd8941521e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:22.620 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.613+0000 7fd899820700 1 -- 172.21.15.88:0/1756123281 >> v1:172.21.15.151:6789/0 conn(0x7fd894151910 legacy=0x7fd894068650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:11:22.620 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.613+0000 7fd899820700 1 -- 172.21.15.88:0/1756123281 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8940b9ff0 con 0x7fd8940c5660 2023-10-18T16:11:22.620 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.613+0000 7fd899820700 1 --2- 172.21.15.88:0/1756123281 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd8940c5660 0x7fd894151cf0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7fd89406ac10 tx=0x7fd8880090f0).ready entity=mon.0 client_cookie=d6788d452f26d313 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:11:22.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.613+0000 7fd89901f700 1 -- 172.21.15.88:0/1756123281 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd88800b630 con 0x7fd8940c5660 2023-10-18T16:11:22.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.613+0000 7fd89901f700 1 -- 172.21.15.88:0/1756123281 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd88800b790 con 0x7fd8940c5660 2023-10-18T16:11:22.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.613+0000 7fd89901f700 1 -- 172.21.15.88:0/1756123281 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd88800bb10 con 0x7fd8940c5660 2023-10-18T16:11:22.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.614+0000 7fd89c285700 1 -- 172.21.15.88:0/1756123281 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd8940c5660 msgr2=0x7fd894151cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:22.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.614+0000 7fd89c285700 1 --2- 172.21.15.88:0/1756123281 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd8940c5660 0x7fd894151cf0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7fd89406ac10 tx=0x7fd8880090f0).stop 2023-10-18T16:11:22.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.614+0000 7fd89c285700 1 -- 172.21.15.88:0/1756123281 shutdown_connections 2023-10-18T16:11:22.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.614+0000 7fd89c285700 1 --2- 172.21.15.88:0/1756123281 >> v2:172.21.15.151:3300/0 conn(0x7fd8940bf400 0x7fd8941521e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:22.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.614+0000 7fd89c285700 1 --2- 172.21.15.88:0/1756123281 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd8940c5660 0x7fd894151cf0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:22.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.614+0000 7fd89c285700 1 -- 172.21.15.88:0/1756123281 >> 172.21.15.88:0/1756123281 conn(0x7fd8940c4940 msgr2=0x7fd8940c4d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:11:22.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.614+0000 7fd89c285700 1 -- 172.21.15.88:0/1756123281 shutdown_connections 2023-10-18T16:11:22.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.614+0000 7fd89c285700 1 -- 172.21.15.88:0/1756123281 wait complete. 2023-10-18T16:11:22.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.615+0000 7fd89c285700 1 Processor -- start 2023-10-18T16:11:22.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.615+0000 7fd89c285700 1 -- start start 2023-10-18T16:11:22.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.616+0000 7fd89c285700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd8940bf400 0x7fd894149b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:22.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.616+0000 7fd89c285700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd8940c5660 0x7fd89415e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:22.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.616+0000 7fd89c285700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd894151910 0x7fd89415e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:22.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.616+0000 7fd89c285700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd894162020 con 0x7fd8940c5660 2023-10-18T16:11:22.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.616+0000 7fd89c285700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd894161710 con 0x7fd8940bf400 2023-10-18T16:11:22.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.616+0000 7fd89c285700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd894147bf0 con 0x7fd894151910 2023-10-18T16:11:22.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.616+0000 7fd899820700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd8940c5660 0x7fd89415e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:22.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.616+0000 7fd899820700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd8940c5660 0x7fd89415e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59526/0 (socket says 172.21.15.88:59526) 2023-10-18T16:11:22.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.616+0000 7fd899820700 1 -- 172.21.15.88:0/2668345785 learned_addr learned my addr 172.21.15.88:0/2668345785 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:11:22.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.616+0000 7fd89a021700 1 --2- 172.21.15.88:0/2668345785 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd8940bf400 0x7fd894149b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:22.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.616+0000 7fd89a822700 1 --2- 172.21.15.88:0/2668345785 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd894151910 0x7fd89415e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:22.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.616+0000 7fd899820700 1 -- 172.21.15.88:0/2668345785 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd894151910 msgr2=0x7fd89415e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:22.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.616+0000 7fd899820700 1 --2- 172.21.15.88:0/2668345785 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd894151910 0x7fd89415e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:22.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.616+0000 7fd899820700 1 -- 172.21.15.88:0/2668345785 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd8940bf400 msgr2=0x7fd894149b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:22.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.616+0000 7fd899820700 1 --2- 172.21.15.88:0/2668345785 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd8940bf400 0x7fd894149b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:22.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.616+0000 7fd899820700 1 -- 172.21.15.88:0/2668345785 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd888009370 con 0x7fd8940c5660 2023-10-18T16:11:22.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.616+0000 7fd899820700 1 --2- 172.21.15.88:0/2668345785 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd8940c5660 0x7fd89415e040 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7fd888009130 tx=0x7fd888015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:11:22.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.617+0000 7fd8927fc700 1 -- 172.21.15.88:0/2668345785 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd88800b900 con 0x7fd8940c5660 2023-10-18T16:11:22.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.617+0000 7fd8927fc700 1 -- 172.21.15.88:0/2668345785 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd88800ba60 con 0x7fd8940c5660 2023-10-18T16:11:22.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.617+0000 7fd8927fc700 1 -- 172.21.15.88:0/2668345785 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd88801c670 con 0x7fd8940c5660 2023-10-18T16:11:22.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.617+0000 7fd89c285700 1 -- 172.21.15.88:0/2668345785 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8941619d0 con 0x7fd8940c5660 2023-10-18T16:11:22.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.617+0000 7fd89c285700 1 -- 172.21.15.88:0/2668345785 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8941512b0 con 0x7fd8940c5660 2023-10-18T16:11:22.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.618+0000 7fd89c285700 1 -- 172.21.15.88:0/2668345785 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd894161e70 con 0x7fd8940c5660 2023-10-18T16:11:22.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.618+0000 7fd8927fc700 1 -- 172.21.15.88:0/2668345785 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7fd88801c9e0 con 0x7fd8940c5660 2023-10-18T16:11:22.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.618+0000 7fd8927fc700 1 --2- 172.21.15.88:0/2668345785 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fd88005f150 0x7fd880061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:22.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.618+0000 7fd8927fc700 1 -- 172.21.15.88:0/2668345785 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6725+0+0 (secure 0 0 0) 0x7fd88807c0c0 con 0x7fd8940c5660 2023-10-18T16:11:22.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.618+0000 7fd89a021700 1 --2- 172.21.15.88:0/2668345785 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fd88005f150 0x7fd880061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:22.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.619+0000 7fd89a021700 1 --2- 172.21.15.88:0/2668345785 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fd88005f150 0x7fd880061610 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fd88c006fd0 tx=0x7fd88c009250).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:11:22.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.622+0000 7fd8927fc700 1 -- 172.21.15.88:0/2668345785 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7fd88804d130 con 0x7fd8940c5660 2023-10-18T16:11:22.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.822+0000 7fd89c285700 1 -- 172.21.15.88:0/2668345785 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd89406c4e0 con 0x7fd8940c5660 2023-10-18T16:11:22.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.822+0000 7fd8927fc700 1 -- 172.21.15.88:0/2668345785 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7fd88802a0e0 con 0x7fd8940c5660 2023-10-18T16:11:22.826 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:11:22.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.824+0000 7fd89c285700 1 -- 172.21.15.88:0/2668345785 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fd88005f150 msgr2=0x7fd880061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:22.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.824+0000 7fd89c285700 1 --2- 172.21.15.88:0/2668345785 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fd88005f150 0x7fd880061610 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fd88c006fd0 tx=0x7fd88c009250).stop 2023-10-18T16:11:22.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.824+0000 7fd89c285700 1 -- 172.21.15.88:0/2668345785 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd8940c5660 msgr2=0x7fd89415e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:22.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.824+0000 7fd89c285700 1 --2- 172.21.15.88:0/2668345785 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd8940c5660 0x7fd89415e040 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7fd888009130 tx=0x7fd888015000).stop 2023-10-18T16:11:22.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.824+0000 7fd89c285700 1 -- 172.21.15.88:0/2668345785 shutdown_connections 2023-10-18T16:11:22.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.824+0000 7fd89c285700 1 --2- 172.21.15.88:0/2668345785 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fd88005f150 0x7fd880061610 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:22.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.824+0000 7fd89c285700 1 --2- 172.21.15.88:0/2668345785 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd894151910 0x7fd89415e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:22.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.824+0000 7fd89c285700 1 --2- 172.21.15.88:0/2668345785 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd8940bf400 0x7fd894149b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:22.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.824+0000 7fd89c285700 1 --2- 172.21.15.88:0/2668345785 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd8940c5660 0x7fd89415e040 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:22.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.824+0000 7fd89c285700 1 -- 172.21.15.88:0/2668345785 >> 172.21.15.88:0/2668345785 conn(0x7fd8940c4940 msgr2=0x7fd894151cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:11:22.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.824+0000 7fd89c285700 1 -- 172.21.15.88:0/2668345785 shutdown_connections 2023-10-18T16:11:22.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:22.824+0000 7fd89c285700 1 -- 172.21.15.88:0/2668345785 wait complete. 2023-10-18T16:11:22.839 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:11:23.176 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~107s 2023-10-18T16:11:23.177 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~120s 2023-10-18T16:11:23.177 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~153s 2023-10-18T16:11:23.177 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~133s 2023-10-18T16:11:23.177 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~214s 2023-10-18T16:11:23.178 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.f is failed for ~166s 2023-10-18T16:11:25.840 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:11:25.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.957+0000 7fc8bafb4700 1 Processor -- start 2023-10-18T16:11:25.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.957+0000 7fc8bafb4700 1 -- start start 2023-10-18T16:11:25.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.958+0000 7fc8bafb4700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fc8b40bf230 0x7fc8b40a98c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:25.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.958+0000 7fc8bafb4700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8b4063c70 con 0x7fc8b40654b0 2023-10-18T16:11:25.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.958+0000 7fc8bafb4700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8b40bf700 con 0x7fc8b4153aa0 2023-10-18T16:11:25.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.958+0000 7fc8bafb4700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8b4069680 con 0x7fc8b40bf230 2023-10-18T16:11:25.969 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.958+0000 7fc8abfff700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fc8b40654b0 0x7fc8b4066b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:48250/0 (socket says 172.21.15.88:48250) 2023-10-18T16:11:25.969 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.958+0000 7fc8abfff700 1 -- 172.21.15.88:0/2587923162 learned_addr learned my addr 172.21.15.88:0/2587923162 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:11:25.969 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.959+0000 7fc8ab7fe700 1 -- 172.21.15.88:0/2587923162 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1037289720 0 0) 0x7fc8b4063c70 con 0x7fc8b40654b0 2023-10-18T16:11:25.969 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.959+0000 7fc8ab7fe700 1 -- 172.21.15.88:0/2587923162 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc89c0033f0 con 0x7fc8b40654b0 2023-10-18T16:11:25.969 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.959+0000 7fc8b9551700 1 --2- 172.21.15.88:0/2587923162 >> v2:172.21.15.151:3300/0 conn(0x7fc8b40bf230 0x7fc8b40a98c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:25.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.959+0000 7fc8ab7fe700 1 -- 172.21.15.88:0/2587923162 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 153632417 0 0) 0x7fc89c0033f0 con 0x7fc8b40654b0 2023-10-18T16:11:25.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.959+0000 7fc8ab7fe700 1 -- 172.21.15.88:0/2587923162 >> v2:172.21.15.151:3300/0 conn(0x7fc8b40bf230 msgr2=0x7fc8b40a98c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:11:25.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.959+0000 7fc8ab7fe700 1 --2- 172.21.15.88:0/2587923162 >> v2:172.21.15.151:3300/0 conn(0x7fc8b40bf230 0x7fc8b40a98c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:25.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.959+0000 7fc8ab7fe700 1 -- 172.21.15.88:0/2587923162 >> v1:172.21.15.151:6789/0 conn(0x7fc8b4153aa0 legacy=0x7fc8b4153e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:25.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.959+0000 7fc8ab7fe700 1 -- 172.21.15.88:0/2587923162 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8b40690c0 con 0x7fc8b40654b0 2023-10-18T16:11:25.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.959+0000 7fc8ab7fe700 1 -- 172.21.15.88:0/2587923162 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441715075 0 0) 0x7fc8a4001470 con 0x7fc8b40654b0 2023-10-18T16:11:25.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.959+0000 7fc8ab7fe700 1 -- 172.21.15.88:0/2587923162 >> v1:172.21.15.88:6789/0 conn(0x7fc8b40654b0 legacy=0x7fc8b4066b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:25.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.959+0000 7fc8ab7fe700 1 --2- 172.21.15.88:0/2587923162 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc89c0038e0 0x7fc89c003d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:25.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.959+0000 7fc8ab7fe700 1 --2- 172.21.15.88:0/2587923162 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc89c004690 0x7fc89c004b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:25.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.959+0000 7fc8ab7fe700 1 --2- 172.21.15.88:0/2587923162 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc89c005050 0x7fc89c00b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:25.972 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.959+0000 7fc8ab7fe700 1 -- 172.21.15.88:0/2587923162 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b4069680 con 0x7fc89c0038e0 2023-10-18T16:11:25.972 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.959+0000 7fc8ab7fe700 1 -- 172.21.15.88:0/2587923162 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc89c00bf30 con 0x7fc89c005050 2023-10-18T16:11:25.972 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.959+0000 7fc8ab7fe700 1 -- 172.21.15.88:0/2587923162 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc89c00c0f0 con 0x7fc89c004690 2023-10-18T16:11:25.972 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.959+0000 7fc8b8d50700 1 --2- 172.21.15.88:0/2587923162 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc89c0038e0 0x7fc89c003d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:25.972 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.959+0000 7fc8abfff700 1 --2- 172.21.15.88:0/2587923162 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc89c004690 0x7fc89c004b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:25.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.959+0000 7fc8b9551700 1 --2- 172.21.15.88:0/2587923162 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc89c005050 0x7fc89c00b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:25.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.959+0000 7fc8b8d50700 1 -- 172.21.15.88:0/2587923162 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc89c004690 msgr2=0x7fc89c004b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:11:25.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.959+0000 7fc8b8d50700 1 --2- 172.21.15.88:0/2587923162 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc89c004690 0x7fc89c004b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:25.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.959+0000 7fc8b8d50700 1 -- 172.21.15.88:0/2587923162 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc89c005050 msgr2=0x7fc89c00b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:11:25.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.959+0000 7fc8b8d50700 1 --2- 172.21.15.88:0/2587923162 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc89c005050 0x7fc89c00b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:25.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.959+0000 7fc8b9551700 1 -- 172.21.15.88:0/2587923162 reap_dead start 2023-10-18T16:11:25.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.959+0000 7fc8b8d50700 1 -- 172.21.15.88:0/2587923162 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc89c0033f0 con 0x7fc89c0038e0 2023-10-18T16:11:25.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.959+0000 7fc8abfff700 1 --2- 172.21.15.88:0/2587923162 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc89c004690 0x7fc89c004b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:11:25.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.959+0000 7fc8b8d50700 1 --2- 172.21.15.88:0/2587923162 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc89c0038e0 0x7fc89c003d00 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7fc8b0001e60 tx=0x7fc8b0007f30).ready entity=mon.0 client_cookie=9acfd0a517d8e77 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:11:25.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.960+0000 7fc8ab7fe700 1 -- 172.21.15.88:0/2587923162 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8b0006a20 con 0x7fc89c0038e0 2023-10-18T16:11:25.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.960+0000 7fc8ab7fe700 1 -- 172.21.15.88:0/2587923162 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8b0006b80 con 0x7fc89c0038e0 2023-10-18T16:11:25.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.960+0000 7fc8ab7fe700 1 -- 172.21.15.88:0/2587923162 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8b000c6a0 con 0x7fc89c0038e0 2023-10-18T16:11:25.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.960+0000 7fc8bafb4700 1 -- 172.21.15.88:0/2587923162 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc89c0038e0 msgr2=0x7fc89c003d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:25.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.960+0000 7fc8bafb4700 1 --2- 172.21.15.88:0/2587923162 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc89c0038e0 0x7fc89c003d00 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7fc8b0001e60 tx=0x7fc8b0007f30).stop 2023-10-18T16:11:25.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.960+0000 7fc8bafb4700 1 -- 172.21.15.88:0/2587923162 shutdown_connections 2023-10-18T16:11:25.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.960+0000 7fc8bafb4700 1 --2- 172.21.15.88:0/2587923162 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc89c0038e0 0x7fc89c003d00 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:25.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.960+0000 7fc8bafb4700 1 -- 172.21.15.88:0/2587923162 >> 172.21.15.88:0/2587923162 conn(0x7fc8b40c4890 msgr2=0x7fc8b40c4ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:11:25.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.960+0000 7fc8bafb4700 1 -- 172.21.15.88:0/2587923162 shutdown_connections 2023-10-18T16:11:25.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.960+0000 7fc8bafb4700 1 -- 172.21.15.88:0/2587923162 wait complete. 2023-10-18T16:11:25.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.961+0000 7fc8bafb4700 1 Processor -- start 2023-10-18T16:11:25.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.961+0000 7fc8bafb4700 1 -- start start 2023-10-18T16:11:25.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.962+0000 7fc8bafb4700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc89c0038e0 0x7fc8b414bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:25.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.962+0000 7fc8bafb4700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc8b40b9900 0x7fc8b40bc4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:25.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.962+0000 7fc8bafb4700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8b40be3c0 0x7fc8b4157560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:25.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.962+0000 7fc8bafb4700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b40bec70 con 0x7fc8b40be3c0 2023-10-18T16:11:25.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.962+0000 7fc8bafb4700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b414fab0 con 0x7fc89c0038e0 2023-10-18T16:11:25.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.962+0000 7fc8bafb4700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b40bc070 con 0x7fc8b40b9900 2023-10-18T16:11:25.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.962+0000 7fc8b9551700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8b40be3c0 0x7fc8b4157560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:25.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.962+0000 7fc8b9551700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8b40be3c0 0x7fc8b4157560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59540/0 (socket says 172.21.15.88:59540) 2023-10-18T16:11:25.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.962+0000 7fc8b9551700 1 -- 172.21.15.88:0/751107994 learned_addr learned my addr 172.21.15.88:0/751107994 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:11:25.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.962+0000 7fc8b8d50700 1 --2- 172.21.15.88:0/751107994 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc89c0038e0 0x7fc8b414bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:25.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.962+0000 7fc8abfff700 1 --2- 172.21.15.88:0/751107994 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc8b40b9900 0x7fc8b40bc4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:25.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.963+0000 7fc8b9551700 1 -- 172.21.15.88:0/751107994 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc8b40b9900 msgr2=0x7fc8b40bc4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:25.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.963+0000 7fc8b9551700 1 --2- 172.21.15.88:0/751107994 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc8b40b9900 0x7fc8b40bc4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:25.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.963+0000 7fc8b9551700 1 -- 172.21.15.88:0/751107994 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc89c0038e0 msgr2=0x7fc8b414bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:25.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.963+0000 7fc8b9551700 1 --2- 172.21.15.88:0/751107994 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc89c0038e0 0x7fc8b414bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:25.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.963+0000 7fc8b9551700 1 -- 172.21.15.88:0/751107994 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8b4150a70 con 0x7fc8b40be3c0 2023-10-18T16:11:25.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.963+0000 7fc8b9551700 1 --2- 172.21.15.88:0/751107994 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8b40be3c0 0x7fc8b4157560 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7fc8b40681f0 tx=0x7fc8ac008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:11:25.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.963+0000 7fc8a8ff9700 1 -- 172.21.15.88:0/751107994 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8ac012610 con 0x7fc8b40be3c0 2023-10-18T16:11:25.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.963+0000 7fc8a8ff9700 1 -- 172.21.15.88:0/751107994 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8ac012770 con 0x7fc8b40be3c0 2023-10-18T16:11:25.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.963+0000 7fc8bafb4700 1 -- 172.21.15.88:0/751107994 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8b41501d0 con 0x7fc8b40be3c0 2023-10-18T16:11:25.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.963+0000 7fc8a8ff9700 1 -- 172.21.15.88:0/751107994 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8ac012b10 con 0x7fc8b40be3c0 2023-10-18T16:11:25.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.963+0000 7fc8bafb4700 1 -- 172.21.15.88:0/751107994 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8b4150790 con 0x7fc8b40be3c0 2023-10-18T16:11:25.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.964+0000 7fc8bafb4700 1 -- 172.21.15.88:0/751107994 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8b40bc1e0 con 0x7fc8b40be3c0 2023-10-18T16:11:25.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.964+0000 7fc8a8ff9700 1 -- 172.21.15.88:0/751107994 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7fc8ac014470 con 0x7fc8b40be3c0 2023-10-18T16:11:25.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.965+0000 7fc8a8ff9700 1 --2- 172.21.15.88:0/751107994 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fc8a005f150 0x7fc8a0061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:25.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.965+0000 7fc8a8ff9700 1 -- 172.21.15.88:0/751107994 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6725+0+0 (secure 0 0 0) 0x7fc8ac07a250 con 0x7fc8b40be3c0 2023-10-18T16:11:25.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.965+0000 7fc8b8d50700 1 --2- 172.21.15.88:0/751107994 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fc8a005f150 0x7fc8a0061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:25.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.965+0000 7fc8b8d50700 1 --2- 172.21.15.88:0/751107994 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fc8a005f150 0x7fc8a0061610 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fc8b0001630 tx=0x7fc8b0015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:11:25.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:25.969+0000 7fc8a8ff9700 1 -- 172.21.15.88:0/751107994 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7fc8b40bc1e0 con 0x7fc8b40be3c0 2023-10-18T16:11:26.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:26.167+0000 7fc8bafb4700 1 -- 172.21.15.88:0/751107994 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc8b40bc1e0 con 0x7fc8b40be3c0 2023-10-18T16:11:26.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:26.167+0000 7fc8a8ff9700 1 -- 172.21.15.88:0/751107994 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7fc8b40bc1e0 con 0x7fc8b40be3c0 2023-10-18T16:11:26.171 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:11:26.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:26.169+0000 7fc8bafb4700 1 -- 172.21.15.88:0/751107994 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fc8a005f150 msgr2=0x7fc8a0061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:26.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:26.169+0000 7fc8bafb4700 1 --2- 172.21.15.88:0/751107994 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fc8a005f150 0x7fc8a0061610 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fc8b0001630 tx=0x7fc8b0015000).stop 2023-10-18T16:11:26.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:26.169+0000 7fc8bafb4700 1 -- 172.21.15.88:0/751107994 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8b40be3c0 msgr2=0x7fc8b4157560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:26.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:26.169+0000 7fc8bafb4700 1 --2- 172.21.15.88:0/751107994 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8b40be3c0 0x7fc8b4157560 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7fc8b40681f0 tx=0x7fc8ac008040).stop 2023-10-18T16:11:26.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:26.169+0000 7fc8bafb4700 1 -- 172.21.15.88:0/751107994 shutdown_connections 2023-10-18T16:11:26.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:26.169+0000 7fc8bafb4700 1 --2- 172.21.15.88:0/751107994 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fc8a005f150 0x7fc8a0061610 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:26.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:26.169+0000 7fc8bafb4700 1 --2- 172.21.15.88:0/751107994 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8b40be3c0 0x7fc8b4157560 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:26.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:26.169+0000 7fc8bafb4700 1 --2- 172.21.15.88:0/751107994 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc89c0038e0 0x7fc8b414bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:26.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:26.169+0000 7fc8bafb4700 1 --2- 172.21.15.88:0/751107994 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc8b40b9900 0x7fc8b40bc4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:26.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:26.169+0000 7fc8bafb4700 1 -- 172.21.15.88:0/751107994 >> 172.21.15.88:0/751107994 conn(0x7fc8b40c4890 msgr2=0x7fc8b414cf80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:11:26.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:26.169+0000 7fc8bafb4700 1 -- 172.21.15.88:0/751107994 shutdown_connections 2023-10-18T16:11:26.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:26.169+0000 7fc8bafb4700 1 -- 172.21.15.88:0/751107994 wait complete. 2023-10-18T16:11:26.184 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:11:29.185 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:11:29.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.307+0000 7f19f1b95700 1 Processor -- start 2023-10-18T16:11:29.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.307+0000 7f19f1b95700 1 -- start start 2023-10-18T16:11:29.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.308+0000 7f19f1b95700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f19ec0a9050 0x7f19ec153ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:29.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.308+0000 7f19f1b95700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19ec064150 con 0x7f19ec14d8c0 2023-10-18T16:11:29.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.308+0000 7f19f1b95700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19ec153a40 con 0x7f19ec065360 2023-10-18T16:11:29.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.308+0000 7f19f1b95700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19ec063e70 con 0x7f19ec0a9050 2023-10-18T16:11:29.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.308+0000 7f19eb7fe700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f19ec14d8c0 0x7f19ec064440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:42110/0 (socket says 172.21.15.88:42110) 2023-10-18T16:11:29.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.308+0000 7f19eb7fe700 1 -- 172.21.15.88:0/4042751386 learned_addr learned my addr 172.21.15.88:0/4042751386 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:11:29.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.308+0000 7f19e2ffd700 1 --2- 172.21.15.88:0/4042751386 >> v2:172.21.15.151:3300/0 conn(0x7f19ec0a9050 0x7f19ec153ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:29.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.308+0000 7f19eaffd700 1 -- 172.21.15.88:0/4042751386 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1803239619 0 0) 0x7f19ec064150 con 0x7f19ec14d8c0 2023-10-18T16:11:29.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.308+0000 7f19eaffd700 1 -- 172.21.15.88:0/4042751386 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19d80033f0 con 0x7f19ec14d8c0 2023-10-18T16:11:29.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.308+0000 7f19eaffd700 1 -- 172.21.15.88:0/4042751386 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 211826756 0 0) 0x7f19d80033f0 con 0x7f19ec14d8c0 2023-10-18T16:11:29.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.308+0000 7f19eaffd700 1 -- 172.21.15.88:0/4042751386 >> v2:172.21.15.151:3300/0 conn(0x7f19ec0a9050 msgr2=0x7f19ec153ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:11:29.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.308+0000 7f19eaffd700 1 --2- 172.21.15.88:0/4042751386 >> v2:172.21.15.151:3300/0 conn(0x7f19ec0a9050 0x7f19ec153ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:29.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.308+0000 7f19eaffd700 1 -- 172.21.15.88:0/4042751386 >> v1:172.21.15.151:6789/0 conn(0x7f19ec065360 legacy=0x7f19ec15acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:29.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.308+0000 7f19eaffd700 1 -- 172.21.15.88:0/4042751386 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19ec0c72c0 con 0x7f19ec14d8c0 2023-10-18T16:11:29.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.308+0000 7f19eaffd700 1 -- 172.21.15.88:0/4042751386 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441715075 0 0) 0x7f19dc001470 con 0x7f19ec14d8c0 2023-10-18T16:11:29.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.308+0000 7f19eaffd700 1 -- 172.21.15.88:0/4042751386 >> v1:172.21.15.88:6789/0 conn(0x7f19ec14d8c0 legacy=0x7f19ec064440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:29.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.308+0000 7f19eaffd700 1 --2- 172.21.15.88:0/4042751386 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f19d80038e0 0x7f19d8003d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:29.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.308+0000 7f19eaffd700 1 --2- 172.21.15.88:0/4042751386 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f19d8004690 0x7f19d8004b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:29.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.308+0000 7f19eaffd700 1 --2- 172.21.15.88:0/4042751386 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f19d8005050 0x7f19d800b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:29.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.308+0000 7f19eaffd700 1 -- 172.21.15.88:0/4042751386 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19ec063e70 con 0x7f19d8004690 2023-10-18T16:11:29.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.308+0000 7f19eaffd700 1 -- 172.21.15.88:0/4042751386 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19d800bfe0 con 0x7f19d80038e0 2023-10-18T16:11:29.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.309+0000 7f19eaffd700 1 -- 172.21.15.88:0/4042751386 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19d800c1a0 con 0x7f19d8005050 2023-10-18T16:11:29.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.309+0000 7f19e2ffd700 1 --2- 172.21.15.88:0/4042751386 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f19d8004690 0x7f19d8004b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:29.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.309+0000 7f19eb7fe700 1 --2- 172.21.15.88:0/4042751386 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f19d80038e0 0x7f19d8003d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:29.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.309+0000 7f19ebfff700 1 --2- 172.21.15.88:0/4042751386 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f19d8005050 0x7f19d800b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:29.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.309+0000 7f19e2ffd700 1 -- 172.21.15.88:0/4042751386 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f19d8005050 msgr2=0x7f19d800b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:11:29.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.309+0000 7f19e2ffd700 1 --2- 172.21.15.88:0/4042751386 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f19d8005050 0x7f19d800b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:29.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.309+0000 7f19e2ffd700 1 -- 172.21.15.88:0/4042751386 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f19d80038e0 msgr2=0x7f19d8003d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:11:29.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.309+0000 7f19e2ffd700 1 --2- 172.21.15.88:0/4042751386 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f19d80038e0 0x7f19d8003d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:29.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.309+0000 7f19e2ffd700 1 -- 172.21.15.88:0/4042751386 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19d80033f0 con 0x7f19d8004690 2023-10-18T16:11:29.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.309+0000 7f19ebfff700 1 --2- 172.21.15.88:0/4042751386 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f19d8005050 0x7f19d800b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:11:29.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.309+0000 7f19ebfff700 1 -- 172.21.15.88:0/4042751386 reap_dead start 2023-10-18T16:11:29.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.309+0000 7f19e2ffd700 1 --2- 172.21.15.88:0/4042751386 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f19d8004690 0x7f19d8004b10 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f19d4006d80 tx=0x7f19d4009060).ready entity=mon.0 client_cookie=2c501d844e43c4e8 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:11:29.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.309+0000 7f19eaffd700 1 -- 172.21.15.88:0/4042751386 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d4009e90 con 0x7f19d8004690 2023-10-18T16:11:29.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.309+0000 7f19eaffd700 1 -- 172.21.15.88:0/4042751386 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19d400e640 con 0x7f19d8004690 2023-10-18T16:11:29.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.309+0000 7f19eaffd700 1 -- 172.21.15.88:0/4042751386 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d400ea00 con 0x7f19d8004690 2023-10-18T16:11:29.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.309+0000 7f19f1b95700 1 -- 172.21.15.88:0/4042751386 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f19d8004690 msgr2=0x7f19d8004b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:29.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.309+0000 7f19f1b95700 1 --2- 172.21.15.88:0/4042751386 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f19d8004690 0x7f19d8004b10 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f19d4006d80 tx=0x7f19d4009060).stop 2023-10-18T16:11:29.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.310+0000 7f19f1b95700 1 -- 172.21.15.88:0/4042751386 shutdown_connections 2023-10-18T16:11:29.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.310+0000 7f19f1b95700 1 --2- 172.21.15.88:0/4042751386 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f19d8004690 0x7f19d8004b10 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:29.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.310+0000 7f19f1b95700 1 -- 172.21.15.88:0/4042751386 >> 172.21.15.88:0/4042751386 conn(0x7f19ec06a810 msgr2=0x7f19ec0c6a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:11:29.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.310+0000 7f19f1b95700 1 -- 172.21.15.88:0/4042751386 shutdown_connections 2023-10-18T16:11:29.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.310+0000 7f19f1b95700 1 -- 172.21.15.88:0/4042751386 wait complete. 2023-10-18T16:11:29.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.311+0000 7f19f1b95700 1 Processor -- start 2023-10-18T16:11:29.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.311+0000 7f19f1b95700 1 -- start start 2023-10-18T16:11:29.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.312+0000 7f19f1b95700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f19d8004690 0x7f19ec0bb860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:29.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.312+0000 7f19f1b95700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f19ec0bd580 0x7f19ec163fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:29.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.312+0000 7f19f1b95700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f19ec0bbd50 0x7f19ec1644d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:29.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.312+0000 7f19f1b95700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19ec0bdf00 con 0x7f19d8004690 2023-10-18T16:11:29.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.312+0000 7f19f1b95700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19ec0ac6f0 con 0x7f19ec0bd580 2023-10-18T16:11:29.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.312+0000 7f19f1b95700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19ec0be1c0 con 0x7f19ec0bbd50 2023-10-18T16:11:29.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.312+0000 7f19eb7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f19d8004690 0x7f19ec0bb860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:29.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.312+0000 7f19eb7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f19d8004690 0x7f19ec0bb860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41998/0 (socket says 172.21.15.88:41998) 2023-10-18T16:11:29.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.312+0000 7f19eb7fe700 1 -- 172.21.15.88:0/3160944776 learned_addr learned my addr 172.21.15.88:0/3160944776 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:11:29.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.312+0000 7f19e2ffd700 1 --2- 172.21.15.88:0/3160944776 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f19ec0bd580 0x7f19ec163fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:29.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.312+0000 7f19ebfff700 1 --2- 172.21.15.88:0/3160944776 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f19ec0bbd50 0x7f19ec1644d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:29.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.312+0000 7f19eb7fe700 1 -- 172.21.15.88:0/3160944776 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f19ec0bbd50 msgr2=0x7f19ec1644d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:29.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.312+0000 7f19eb7fe700 1 --2- 172.21.15.88:0/3160944776 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f19ec0bbd50 0x7f19ec1644d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:29.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.312+0000 7f19eb7fe700 1 -- 172.21.15.88:0/3160944776 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f19ec0bd580 msgr2=0x7f19ec163fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:29.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.312+0000 7f19eb7fe700 1 --2- 172.21.15.88:0/3160944776 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f19ec0bd580 0x7f19ec163fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:29.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.312+0000 7f19eb7fe700 1 -- 172.21.15.88:0/3160944776 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19ec0be4a0 con 0x7f19d8004690 2023-10-18T16:11:29.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.312+0000 7f19eb7fe700 1 --2- 172.21.15.88:0/3160944776 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f19d8004690 0x7f19ec0bb860 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7f19dc0047b0 tx=0x7f19dc001e50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:11:29.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.312+0000 7f19e3fff700 1 -- 172.21.15.88:0/3160944776 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19dc00f5a0 con 0x7f19d8004690 2023-10-18T16:11:29.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.312+0000 7f19e3fff700 1 -- 172.21.15.88:0/3160944776 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19dc00f700 con 0x7f19d8004690 2023-10-18T16:11:29.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.312+0000 7f19f1b95700 1 -- 172.21.15.88:0/3160944776 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19ec0abc10 con 0x7f19d8004690 2023-10-18T16:11:29.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.312+0000 7f19e3fff700 1 -- 172.21.15.88:0/3160944776 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19dc00fa80 con 0x7f19d8004690 2023-10-18T16:11:29.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.312+0000 7f19f1b95700 1 -- 172.21.15.88:0/3160944776 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19ec0ac1d0 con 0x7f19d8004690 2023-10-18T16:11:29.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.313+0000 7f19f1b95700 1 -- 172.21.15.88:0/3160944776 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19ec06c4e0 con 0x7f19d8004690 2023-10-18T16:11:29.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.314+0000 7f19e3fff700 1 -- 172.21.15.88:0/3160944776 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f19dc01d3b0 con 0x7f19d8004690 2023-10-18T16:11:29.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.314+0000 7f19e3fff700 1 --2- 172.21.15.88:0/3160944776 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f19cc05f150 0x7f19cc061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:29.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.314+0000 7f19e3fff700 1 -- 172.21.15.88:0/3160944776 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6725+0+0 (secure 0 0 0) 0x7f19dc079d70 con 0x7f19d8004690 2023-10-18T16:11:29.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.314+0000 7f19e2ffd700 1 --2- 172.21.15.88:0/3160944776 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f19cc05f150 0x7f19cc061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:29.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.315+0000 7f19e2ffd700 1 --2- 172.21.15.88:0/3160944776 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f19cc05f150 0x7f19cc061610 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f19ec084880 tx=0x7f19d401a000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:11:29.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.318+0000 7f19e3fff700 1 -- 172.21.15.88:0/3160944776 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f19dc04ae30 con 0x7f19d8004690 2023-10-18T16:11:29.488 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~114s 2023-10-18T16:11:29.489 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~126s 2023-10-18T16:11:29.489 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~159s 2023-10-18T16:11:29.489 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~139s 2023-10-18T16:11:29.489 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~220s 2023-10-18T16:11:29.489 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.f is failed for ~173s 2023-10-18T16:11:29.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.520+0000 7f19f1b95700 1 -- 172.21.15.88:0/3160944776 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f19ec0c75a0 con 0x7f19d8004690 2023-10-18T16:11:29.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.521+0000 7f19e3fff700 1 -- 172.21.15.88:0/3160944776 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7f19dc019030 con 0x7f19d8004690 2023-10-18T16:11:29.525 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:11:29.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.523+0000 7f19f1b95700 1 -- 172.21.15.88:0/3160944776 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f19cc05f150 msgr2=0x7f19cc061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:29.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.523+0000 7f19f1b95700 1 --2- 172.21.15.88:0/3160944776 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f19cc05f150 0x7f19cc061610 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f19ec084880 tx=0x7f19d401a000).stop 2023-10-18T16:11:29.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.523+0000 7f19f1b95700 1 -- 172.21.15.88:0/3160944776 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f19d8004690 msgr2=0x7f19ec0bb860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:29.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.523+0000 7f19f1b95700 1 --2- 172.21.15.88:0/3160944776 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f19d8004690 0x7f19ec0bb860 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7f19dc0047b0 tx=0x7f19dc001e50).stop 2023-10-18T16:11:29.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.523+0000 7f19f1b95700 1 -- 172.21.15.88:0/3160944776 shutdown_connections 2023-10-18T16:11:29.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.523+0000 7f19f1b95700 1 --2- 172.21.15.88:0/3160944776 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f19cc05f150 0x7f19cc061610 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:29.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.523+0000 7f19f1b95700 1 --2- 172.21.15.88:0/3160944776 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f19ec0bbd50 0x7f19ec1644d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:29.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.523+0000 7f19f1b95700 1 --2- 172.21.15.88:0/3160944776 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f19d8004690 0x7f19ec0bb860 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:29.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.523+0000 7f19f1b95700 1 --2- 172.21.15.88:0/3160944776 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f19ec0bd580 0x7f19ec163fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:29.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.523+0000 7f19f1b95700 1 -- 172.21.15.88:0/3160944776 >> 172.21.15.88:0/3160944776 conn(0x7f19ec06a810 msgr2=0x7f19ec153ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:11:29.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.523+0000 7f19f1b95700 1 -- 172.21.15.88:0/3160944776 shutdown_connections 2023-10-18T16:11:29.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:29.523+0000 7f19f1b95700 1 -- 172.21.15.88:0/3160944776 wait complete. 2023-10-18T16:11:29.538 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:11:32.539 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:11:32.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.660+0000 7fc8c7fff700 1 Processor -- start 2023-10-18T16:11:32.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.660+0000 7fc8c7fff700 1 -- start start 2023-10-18T16:11:32.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.661+0000 7fc8c7fff700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fc8c80c6a60 0x7fc8c80c6e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:32.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.661+0000 7fc8c7fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc8c80c2400 0x7fc8c80c2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:32.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.661+0000 7fc8c7fff700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8c8151020 con 0x7fc8c8063d70 2023-10-18T16:11:32.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.661+0000 7fc8c7fff700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8c8150090 con 0x7fc8c80c6a60 2023-10-18T16:11:32.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.661+0000 7fc8c7fff700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c80c2d80 con 0x7fc8c80c2400 2023-10-18T16:11:32.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.661+0000 7fc8c67fc700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fc8c8063d70 0x7fc8c8153660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:42112/0 (socket says 172.21.15.88:42112) 2023-10-18T16:11:32.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.661+0000 7fc8c67fc700 1 -- 172.21.15.88:0/173912868 learned_addr learned my addr 172.21.15.88:0/173912868 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:11:32.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.661+0000 7fc8c77fe700 1 --2- 172.21.15.88:0/173912868 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc8c80c2400 0x7fc8c80c2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:32.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.661+0000 7fc8c6ffd700 1 --2- 172.21.15.88:0/173912868 >> v2:172.21.15.151:3300/0 conn(0x7fc8c80c6a60 0x7fc8c80c6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:32.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.661+0000 7fc8c5ffb700 1 -- 172.21.15.88:0/173912868 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3789170418 0 0) 0x7fc8b0001110 con 0x7fc8c8063d70 2023-10-18T16:11:32.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.661+0000 7fc8c5ffb700 1 -- 172.21.15.88:0/173912868 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8b40033f0 con 0x7fc8c8063d70 2023-10-18T16:11:32.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.662+0000 7fc8c5ffb700 1 -- 172.21.15.88:0/173912868 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 517741079 0 0) 0x7fc8b40033f0 con 0x7fc8c8063d70 2023-10-18T16:11:32.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.662+0000 7fc8c5ffb700 1 -- 172.21.15.88:0/173912868 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc8c80c2400 msgr2=0x7fc8c80c2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:11:32.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.662+0000 7fc8c5ffb700 1 --2- 172.21.15.88:0/173912868 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc8c80c2400 0x7fc8c80c2890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:32.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.662+0000 7fc8c5ffb700 1 -- 172.21.15.88:0/173912868 >> v2:172.21.15.151:3300/0 conn(0x7fc8c80c6a60 msgr2=0x7fc8c80c6e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:11:32.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.662+0000 7fc8c5ffb700 1 --2- 172.21.15.88:0/173912868 >> v2:172.21.15.151:3300/0 conn(0x7fc8c80c6a60 0x7fc8c80c6e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:32.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.662+0000 7fc8c5ffb700 1 -- 172.21.15.88:0/173912868 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8c8150b90 con 0x7fc8c8063d70 2023-10-18T16:11:32.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.662+0000 7fc8c5ffb700 1 -- 172.21.15.88:0/173912868 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441715075 0 0) 0x7fc8b0001600 con 0x7fc8c8063d70 2023-10-18T16:11:32.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.662+0000 7fc8c5ffb700 1 -- 172.21.15.88:0/173912868 >> v1:172.21.15.88:6789/0 conn(0x7fc8c8063d70 legacy=0x7fc8c8153660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:32.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.662+0000 7fc8c5ffb700 1 --2- 172.21.15.88:0/173912868 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc8b4003910 0x7fc8b4003d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:32.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.662+0000 7fc8c5ffb700 1 --2- 172.21.15.88:0/173912868 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8b4004690 0x7fc8b4004b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:32.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.662+0000 7fc8c5ffb700 1 --2- 172.21.15.88:0/173912868 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc8c80c2400 0x7fc8b4005000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:32.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.662+0000 7fc8c5ffb700 1 -- 172.21.15.88:0/173912868 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b4005540 con 0x7fc8b4004690 2023-10-18T16:11:32.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.662+0000 7fc8c5ffb700 1 -- 172.21.15.88:0/173912868 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b400b910 con 0x7fc8b4003910 2023-10-18T16:11:32.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.662+0000 7fc8c5ffb700 1 -- 172.21.15.88:0/173912868 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b400bad0 con 0x7fc8c80c2400 2023-10-18T16:11:32.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.662+0000 7fc8c67fc700 1 --2- 172.21.15.88:0/173912868 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8b4004690 0x7fc8b4004b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:32.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.662+0000 7fc8c6ffd700 1 --2- 172.21.15.88:0/173912868 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc8b4003910 0x7fc8b4003d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:32.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.662+0000 7fc8c77fe700 1 --2- 172.21.15.88:0/173912868 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc8c80c2400 0x7fc8b4005000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:32.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.662+0000 7fc8c67fc700 1 -- 172.21.15.88:0/173912868 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc8c80c2400 msgr2=0x7fc8b4005000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:11:32.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.662+0000 7fc8c67fc700 1 --2- 172.21.15.88:0/173912868 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc8c80c2400 0x7fc8b4005000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:32.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.662+0000 7fc8c67fc700 1 -- 172.21.15.88:0/173912868 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc8b4003910 msgr2=0x7fc8b4003d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:11:32.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.662+0000 7fc8c67fc700 1 --2- 172.21.15.88:0/173912868 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc8b4003910 0x7fc8b4003d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:32.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.662+0000 7fc8c67fc700 1 -- 172.21.15.88:0/173912868 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8b40033f0 con 0x7fc8b4004690 2023-10-18T16:11:32.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.662+0000 7fc8c67fc700 1 --2- 172.21.15.88:0/173912868 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8b4004690 0x7fc8b4004b10 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7fc8b0014bd0 tx=0x7fc8c80c3210).ready entity=mon.0 client_cookie=30fb08259dbc40e1 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:11:32.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.662+0000 7fc8c5ffb700 1 -- 172.21.15.88:0/173912868 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8b0007c00 con 0x7fc8b4004690 2023-10-18T16:11:32.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.662+0000 7fc8c5ffb700 1 -- 172.21.15.88:0/173912868 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8b0007110 con 0x7fc8b4004690 2023-10-18T16:11:32.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.662+0000 7fc8c5ffb700 1 -- 172.21.15.88:0/173912868 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8b0007490 con 0x7fc8b4004690 2023-10-18T16:11:32.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.663+0000 7fc8c7fff700 1 -- 172.21.15.88:0/173912868 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8b4004690 msgr2=0x7fc8b4004b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:32.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.663+0000 7fc8c7fff700 1 --2- 172.21.15.88:0/173912868 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8b4004690 0x7fc8b4004b10 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7fc8b0014bd0 tx=0x7fc8c80c3210).stop 2023-10-18T16:11:32.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.663+0000 7fc8c77fe700 1 -- 172.21.15.88:0/173912868 reap_dead start 2023-10-18T16:11:32.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.663+0000 7fc8c7fff700 1 -- 172.21.15.88:0/173912868 shutdown_connections 2023-10-18T16:11:32.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.663+0000 7fc8c7fff700 1 -- 172.21.15.88:0/173912868 >> 172.21.15.88:0/173912868 conn(0x7fc8c805e050 msgr2=0x7fc8c805e460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:11:32.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.663+0000 7fc8c7fff700 1 -- 172.21.15.88:0/173912868 shutdown_connections 2023-10-18T16:11:32.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.663+0000 7fc8c7fff700 1 -- 172.21.15.88:0/173912868 wait complete. 2023-10-18T16:11:32.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.664+0000 7fc8c7fff700 1 Processor -- start 2023-10-18T16:11:32.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.664+0000 7fc8c7fff700 1 -- start start 2023-10-18T16:11:32.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.665+0000 7fc8c7fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8b4004690 0x7fc8c8153140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:32.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.665+0000 7fc8c7fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc8c8153ba0 0x7fc8c814a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:32.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.665+0000 7fc8c7fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc8c814f440 0x7fc8c8145b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:32.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.665+0000 7fc8c7fff700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c8151450 con 0x7fc8b4004690 2023-10-18T16:11:32.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.665+0000 7fc8c7fff700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c8151cb0 con 0x7fc8c8153ba0 2023-10-18T16:11:32.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.665+0000 7fc8c7fff700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c8154090 con 0x7fc8c814f440 2023-10-18T16:11:32.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.665+0000 7fc8c6ffd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8b4004690 0x7fc8c8153140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:32.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.665+0000 7fc8c6ffd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8b4004690 0x7fc8c8153140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42010/0 (socket says 172.21.15.88:42010) 2023-10-18T16:11:32.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.665+0000 7fc8c6ffd700 1 -- 172.21.15.88:0/2464863782 learned_addr learned my addr 172.21.15.88:0/2464863782 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:11:32.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.665+0000 7fc8c67fc700 1 --2- 172.21.15.88:0/2464863782 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc8c8153ba0 0x7fc8c814a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:32.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.665+0000 7fc8c77fe700 1 --2- 172.21.15.88:0/2464863782 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc8c814f440 0x7fc8c8145b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:32.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.665+0000 7fc8c6ffd700 1 -- 172.21.15.88:0/2464863782 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc8c814f440 msgr2=0x7fc8c8145b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:32.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.665+0000 7fc8c6ffd700 1 --2- 172.21.15.88:0/2464863782 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc8c814f440 0x7fc8c8145b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:32.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.665+0000 7fc8c6ffd700 1 -- 172.21.15.88:0/2464863782 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc8c8153ba0 msgr2=0x7fc8c814a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:32.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.665+0000 7fc8c6ffd700 1 --2- 172.21.15.88:0/2464863782 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc8c8153ba0 0x7fc8c814a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:32.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.665+0000 7fc8c6ffd700 1 -- 172.21.15.88:0/2464863782 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8c8154370 con 0x7fc8b4004690 2023-10-18T16:11:32.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.666+0000 7fc8c6ffd700 1 --2- 172.21.15.88:0/2464863782 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8b4004690 0x7fc8c8153140 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7fc8b8011fb0 tx=0x7fc8b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:11:32.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.666+0000 7fc8beffd700 1 -- 172.21.15.88:0/2464863782 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8b800e040 con 0x7fc8b4004690 2023-10-18T16:11:32.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.666+0000 7fc8beffd700 1 -- 172.21.15.88:0/2464863782 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8b800f040 con 0x7fc8b4004690 2023-10-18T16:11:32.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.666+0000 7fc8c7fff700 1 -- 172.21.15.88:0/2464863782 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8c8151660 con 0x7fc8b4004690 2023-10-18T16:11:32.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.666+0000 7fc8beffd700 1 -- 172.21.15.88:0/2464863782 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8b800fed0 con 0x7fc8b4004690 2023-10-18T16:11:32.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.666+0000 7fc8c7fff700 1 -- 172.21.15.88:0/2464863782 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8c814f8c0 con 0x7fc8b4004690 2023-10-18T16:11:32.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.667+0000 7fc8c7fff700 1 -- 172.21.15.88:0/2464863782 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8c80c7240 con 0x7fc8b4004690 2023-10-18T16:11:32.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.667+0000 7fc8beffd700 1 -- 172.21.15.88:0/2464863782 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7fc8b8015660 con 0x7fc8b4004690 2023-10-18T16:11:32.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.667+0000 7fc8beffd700 1 --2- 172.21.15.88:0/2464863782 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fc8a805f150 0x7fc8a8061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:32.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.668+0000 7fc8beffd700 1 -- 172.21.15.88:0/2464863782 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6725+0+0 (secure 0 0 0) 0x7fc8b807b4a0 con 0x7fc8b4004690 2023-10-18T16:11:32.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.668+0000 7fc8c67fc700 1 --2- 172.21.15.88:0/2464863782 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fc8a805f150 0x7fc8a8061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:32.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.668+0000 7fc8c67fc700 1 --2- 172.21.15.88:0/2464863782 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fc8a805f150 0x7fc8a8061610 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fc8c8153fc0 tx=0x7fc8b0007000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:11:32.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.672+0000 7fc8beffd700 1 -- 172.21.15.88:0/2464863782 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7fc8b804c450 con 0x7fc8b4004690 2023-10-18T16:11:32.876 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.872+0000 7fc8c7fff700 1 -- 172.21.15.88:0/2464863782 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc8c814fb10 con 0x7fc8b4004690 2023-10-18T16:11:33.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.872+0000 7fc8beffd700 1 -- 172.21.15.88:0/2464863782 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7fc8b804fa70 con 0x7fc8b4004690 2023-10-18T16:11:33.476 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:11:33.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.873+0000 7fc8c7fff700 1 -- 172.21.15.88:0/2464863782 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fc8a805f150 msgr2=0x7fc8a8061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:33.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.873+0000 7fc8c7fff700 1 --2- 172.21.15.88:0/2464863782 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fc8a805f150 0x7fc8a8061610 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fc8c8153fc0 tx=0x7fc8b0007000).stop 2023-10-18T16:11:33.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.873+0000 7fc8c7fff700 1 -- 172.21.15.88:0/2464863782 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8b4004690 msgr2=0x7fc8c8153140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:33.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.873+0000 7fc8c7fff700 1 --2- 172.21.15.88:0/2464863782 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8b4004690 0x7fc8c8153140 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7fc8b8011fb0 tx=0x7fc8b800b040).stop 2023-10-18T16:11:33.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.874+0000 7fc8c7fff700 1 -- 172.21.15.88:0/2464863782 shutdown_connections 2023-10-18T16:11:33.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.874+0000 7fc8c7fff700 1 --2- 172.21.15.88:0/2464863782 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fc8a805f150 0x7fc8a8061610 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:33.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.874+0000 7fc8c7fff700 1 --2- 172.21.15.88:0/2464863782 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc8c814f440 0x7fc8c8145b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:33.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.874+0000 7fc8c7fff700 1 --2- 172.21.15.88:0/2464863782 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8b4004690 0x7fc8c8153140 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:33.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.874+0000 7fc8c7fff700 1 --2- 172.21.15.88:0/2464863782 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc8c8153ba0 0x7fc8c814a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:33.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.874+0000 7fc8c7fff700 1 -- 172.21.15.88:0/2464863782 >> 172.21.15.88:0/2464863782 conn(0x7fc8c805e050 msgr2=0x7fc8c80c56e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:11:33.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.874+0000 7fc8c7fff700 1 -- 172.21.15.88:0/2464863782 shutdown_connections 2023-10-18T16:11:33.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:32.874+0000 7fc8c7fff700 1 -- 172.21.15.88:0/2464863782 wait complete. 2023-10-18T16:11:33.479 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:11:35.799 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~120s 2023-10-18T16:11:35.799 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~133s 2023-10-18T16:11:35.799 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~166s 2023-10-18T16:11:35.799 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~146s 2023-10-18T16:11:35.800 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~227s 2023-10-18T16:11:35.800 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.f is failed for ~179s 2023-10-18T16:11:36.481 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:11:36.608 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.604+0000 7fa164abc700 1 Processor -- start 2023-10-18T16:11:36.608 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.604+0000 7fa164abc700 1 -- start start 2023-10-18T16:11:36.610 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.605+0000 7fa164abc700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa16014d8c0 0x7fa160064440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:36.610 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.605+0000 7fa164abc700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa160065360 0x7fa160064930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:36.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.605+0000 7fa164abc700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa160064150 con 0x7fa1600a9050 2023-10-18T16:11:36.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.605+0000 7fa164abc700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa160153a40 con 0x7fa160065360 2023-10-18T16:11:36.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.605+0000 7fa164abc700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa160063e70 con 0x7fa16014d8c0 2023-10-18T16:11:36.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.605+0000 7fa15ed9d700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa160065360 0x7fa160064930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:36.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.605+0000 7fa15ed9d700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa160065360 0x7fa160064930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42018/0 (socket says 172.21.15.88:42018) 2023-10-18T16:11:36.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.605+0000 7fa15ed9d700 1 -- 172.21.15.88:0/2526212963 learned_addr learned my addr 172.21.15.88:0/2526212963 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:11:36.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.605+0000 7fa15e59c700 1 --2- 172.21.15.88:0/2526212963 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa16014d8c0 0x7fa160064440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:36.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.605+0000 7fa15ed9d700 1 -- 172.21.15.88:0/2526212963 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa16014d8c0 msgr2=0x7fa160064440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:11:36.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.605+0000 7fa15ed9d700 1 --2- 172.21.15.88:0/2526212963 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa16014d8c0 0x7fa160064440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:36.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.605+0000 7fa15ed9d700 1 -- 172.21.15.88:0/2526212963 >> v1:172.21.15.88:6789/0 conn(0x7fa1600a9050 legacy=0x7fa160153ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:36.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.605+0000 7fa15ed9d700 1 -- 172.21.15.88:0/2526212963 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1600c72c0 con 0x7fa160065360 2023-10-18T16:11:36.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.605+0000 7fa15dd9b700 1 -- 172.21.15.88:0/2526212963 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3618144737 0 0) 0x7fa160064150 con 0x7fa1600a9050 2023-10-18T16:11:36.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.606+0000 7fa15ed9d700 1 --2- 172.21.15.88:0/2526212963 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa160065360 0x7fa160064930 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7fa158009f20 tx=0x7fa158009840).ready entity=mon.0 client_cookie=770f78c95549c91e server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:11:36.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.606+0000 7fa15dd9b700 1 -- 172.21.15.88:0/2526212963 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa15800e070 con 0x7fa160065360 2023-10-18T16:11:36.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.606+0000 7fa15dd9b700 1 -- 172.21.15.88:0/2526212963 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa15800ba80 con 0x7fa160065360 2023-10-18T16:11:36.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.606+0000 7fa15dd9b700 1 -- 172.21.15.88:0/2526212963 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa15800be00 con 0x7fa160065360 2023-10-18T16:11:36.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.606+0000 7fa164abc700 1 -- 172.21.15.88:0/2526212963 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa160065360 msgr2=0x7fa160064930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:36.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.606+0000 7fa164abc700 1 --2- 172.21.15.88:0/2526212963 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa160065360 0x7fa160064930 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7fa158009f20 tx=0x7fa158009840).stop 2023-10-18T16:11:36.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.606+0000 7fa164abc700 1 -- 172.21.15.88:0/2526212963 shutdown_connections 2023-10-18T16:11:36.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.606+0000 7fa164abc700 1 --2- 172.21.15.88:0/2526212963 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa160065360 0x7fa160064930 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:36.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.606+0000 7fa164abc700 1 --2- 172.21.15.88:0/2526212963 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa16014d8c0 0x7fa160064440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:36.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.606+0000 7fa164abc700 1 -- 172.21.15.88:0/2526212963 >> 172.21.15.88:0/2526212963 conn(0x7fa16006a810 msgr2=0x7fa1600c6a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:11:36.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.606+0000 7fa164abc700 1 -- 172.21.15.88:0/2526212963 shutdown_connections 2023-10-18T16:11:36.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.606+0000 7fa164abc700 1 -- 172.21.15.88:0/2526212963 wait complete. 2023-10-18T16:11:36.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.607+0000 7fa164abc700 1 Processor -- start 2023-10-18T16:11:36.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.608+0000 7fa164abc700 1 -- start start 2023-10-18T16:11:36.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.608+0000 7fa164abc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa160065360 0x7fa1600c60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:36.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.608+0000 7fa164abc700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa1600a9050 0x7fa1601635c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:36.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.608+0000 7fa164abc700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa16014d8c0 0x7fa160163ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:36.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.608+0000 7fa164abc700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1600bb360 con 0x7fa16014d8c0 2023-10-18T16:11:36.620 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.608+0000 7fa164abc700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa160152c50 con 0x7fa160065360 2023-10-18T16:11:36.620 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.608+0000 7fa164abc700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1600ac050 con 0x7fa1600a9050 2023-10-18T16:11:36.620 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.608+0000 7fa15ed9d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa16014d8c0 0x7fa160163ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:36.620 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.608+0000 7fa15ed9d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa16014d8c0 0x7fa160163ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42026/0 (socket says 172.21.15.88:42026) 2023-10-18T16:11:36.620 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.608+0000 7fa15ed9d700 1 -- 172.21.15.88:0/1184653552 learned_addr learned my addr 172.21.15.88:0/1184653552 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:11:36.620 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.609+0000 7fa15e59c700 1 --2- 172.21.15.88:0/1184653552 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa160065360 0x7fa1600c60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:36.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.609+0000 7fa155d9b700 1 --2- 172.21.15.88:0/1184653552 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa1600a9050 0x7fa1601635c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:36.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.609+0000 7fa15ed9d700 1 -- 172.21.15.88:0/1184653552 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa1600a9050 msgr2=0x7fa1601635c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:36.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.609+0000 7fa15ed9d700 1 --2- 172.21.15.88:0/1184653552 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa1600a9050 0x7fa1601635c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:36.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.609+0000 7fa15ed9d700 1 -- 172.21.15.88:0/1184653552 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa160065360 msgr2=0x7fa1600c60a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:36.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.609+0000 7fa15ed9d700 1 --2- 172.21.15.88:0/1184653552 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa160065360 0x7fa1600c60a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:36.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.609+0000 7fa15ed9d700 1 -- 172.21.15.88:0/1184653552 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa158009ae0 con 0x7fa16014d8c0 2023-10-18T16:11:36.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.609+0000 7fa155d9b700 1 --2- 172.21.15.88:0/1184653552 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa1600a9050 0x7fa1601635c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-10-18T16:11:36.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.609+0000 7fa15e59c700 1 --2- 172.21.15.88:0/1184653552 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa160065360 0x7fa1600c60a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-10-18T16:11:36.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.609+0000 7fa15ed9d700 1 --2- 172.21.15.88:0/1184653552 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa16014d8c0 0x7fa160163ab0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7fa15800b6c0 tx=0x7fa158009840).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:11:36.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.609+0000 7fa156ffd700 1 -- 172.21.15.88:0/1184653552 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa158007080 con 0x7fa16014d8c0 2023-10-18T16:11:36.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.609+0000 7fa156ffd700 1 -- 172.21.15.88:0/1184653552 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1580092e0 con 0x7fa16014d8c0 2023-10-18T16:11:36.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.609+0000 7fa164abc700 1 -- 172.21.15.88:0/1184653552 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1600baa30 con 0x7fa16014d8c0 2023-10-18T16:11:36.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.609+0000 7fa156ffd700 1 -- 172.21.15.88:0/1184653552 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa15801e680 con 0x7fa16014d8c0 2023-10-18T16:11:36.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.609+0000 7fa164abc700 1 -- 172.21.15.88:0/1184653552 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1600baff0 con 0x7fa16014d8c0 2023-10-18T16:11:36.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.610+0000 7fa164abc700 1 -- 172.21.15.88:0/1184653552 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1600bb510 con 0x7fa16014d8c0 2023-10-18T16:11:36.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.610+0000 7fa156ffd700 1 -- 172.21.15.88:0/1184653552 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7fa158003040 con 0x7fa16014d8c0 2023-10-18T16:11:36.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.610+0000 7fa156ffd700 1 --2- 172.21.15.88:0/1184653552 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fa14005f120 0x7fa1400615e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:36.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.611+0000 7fa156ffd700 1 -- 172.21.15.88:0/1184653552 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6725+0+0 (secure 0 0 0) 0x7fa15807b8a0 con 0x7fa16014d8c0 2023-10-18T16:11:36.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.611+0000 7fa15e59c700 1 --2- 172.21.15.88:0/1184653552 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fa14005f120 0x7fa1400615e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:36.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.611+0000 7fa15e59c700 1 --2- 172.21.15.88:0/1184653552 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fa14005f120 0x7fa1400615e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fa150009810 tx=0x7fa150008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:11:36.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.615+0000 7fa156ffd700 1 -- 172.21.15.88:0/1184653552 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7fa1600bb510 con 0x7fa16014d8c0 2023-10-18T16:11:36.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.815+0000 7fa164abc700 1 -- 172.21.15.88:0/1184653552 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fa16006c4e0 con 0x7fa16014d8c0 2023-10-18T16:11:36.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.815+0000 7fa156ffd700 1 -- 172.21.15.88:0/1184653552 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7fa158023020 con 0x7fa16014d8c0 2023-10-18T16:11:36.819 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:11:36.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.817+0000 7fa164abc700 1 -- 172.21.15.88:0/1184653552 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fa14005f120 msgr2=0x7fa1400615e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:36.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.817+0000 7fa164abc700 1 --2- 172.21.15.88:0/1184653552 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fa14005f120 0x7fa1400615e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fa150009810 tx=0x7fa150008040).stop 2023-10-18T16:11:36.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.817+0000 7fa164abc700 1 -- 172.21.15.88:0/1184653552 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa16014d8c0 msgr2=0x7fa160163ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:36.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.817+0000 7fa164abc700 1 --2- 172.21.15.88:0/1184653552 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa16014d8c0 0x7fa160163ab0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7fa15800b6c0 tx=0x7fa158009840).stop 2023-10-18T16:11:36.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.817+0000 7fa164abc700 1 -- 172.21.15.88:0/1184653552 shutdown_connections 2023-10-18T16:11:36.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.817+0000 7fa164abc700 1 --2- 172.21.15.88:0/1184653552 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fa14005f120 0x7fa1400615e0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:36.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.817+0000 7fa164abc700 1 --2- 172.21.15.88:0/1184653552 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa16014d8c0 0x7fa160163ab0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:36.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.817+0000 7fa164abc700 1 --2- 172.21.15.88:0/1184653552 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa160065360 0x7fa1600c60a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:36.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.817+0000 7fa164abc700 1 --2- 172.21.15.88:0/1184653552 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa1600a9050 0x7fa1601635c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:36.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.817+0000 7fa164abc700 1 -- 172.21.15.88:0/1184653552 >> 172.21.15.88:0/1184653552 conn(0x7fa16006a810 msgr2=0x7fa160151810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:11:36.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.817+0000 7fa164abc700 1 -- 172.21.15.88:0/1184653552 shutdown_connections 2023-10-18T16:11:36.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:36.817+0000 7fa164abc700 1 -- 172.21.15.88:0/1184653552 wait complete. 2023-10-18T16:11:36.832 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:11:38.712 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-18T16:11:38.715 DEBUG:teuthology.orchestra.run.smithi151:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-18T16:11:39.833 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:11:39.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.955+0000 7f99b9cac700 1 Processor -- start 2023-10-18T16:11:39.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.955+0000 7f99b9cac700 1 -- start start 2023-10-18T16:11:39.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.956+0000 7f99b9cac700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f99b414d8c0 0x7f99b4064440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:39.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.956+0000 7f99b9cac700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f99b40a9050 0x7f99b4064930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:39.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.956+0000 7f99b9cac700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99b4064150 con 0x7f99b4065360 2023-10-18T16:11:39.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.956+0000 7f99b9cac700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99b4153a40 con 0x7f99b414d8c0 2023-10-18T16:11:39.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.956+0000 7f99b9cac700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99b4063e70 con 0x7f99b40a9050 2023-10-18T16:11:39.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.956+0000 7f99b3fff700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f99b4065360 0x7f99b4153ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:49480/0 (socket says 172.21.15.88:49480) 2023-10-18T16:11:39.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.956+0000 7f99b3fff700 1 -- 172.21.15.88:0/759498365 learned_addr learned my addr 172.21.15.88:0/759498365 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:11:39.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.956+0000 7f99b2ffd700 1 --2- 172.21.15.88:0/759498365 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f99b40a9050 0x7f99b4064930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:39.969 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.956+0000 7f99b27fc700 1 -- 172.21.15.88:0/759498365 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1614379722 0 0) 0x7f99b4064150 con 0x7f99b4065360 2023-10-18T16:11:39.969 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.956+0000 7f99b27fc700 1 -- 172.21.15.88:0/759498365 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f999c0033f0 con 0x7f99b4065360 2023-10-18T16:11:39.969 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.956+0000 7f99b37fe700 1 --2- 172.21.15.88:0/759498365 >> v2:172.21.15.151:3300/0 conn(0x7f99b414d8c0 0x7f99b4064440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:39.969 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.957+0000 7f99b27fc700 1 -- 172.21.15.88:0/759498365 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1229474146 0 0) 0x7f999c0033f0 con 0x7f99b4065360 2023-10-18T16:11:39.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.957+0000 7f99b27fc700 1 -- 172.21.15.88:0/759498365 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f99b40a9050 msgr2=0x7f99b4064930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:11:39.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.957+0000 7f99b27fc700 1 --2- 172.21.15.88:0/759498365 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f99b40a9050 0x7f99b4064930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:39.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.957+0000 7f99b27fc700 1 -- 172.21.15.88:0/759498365 >> v2:172.21.15.151:3300/0 conn(0x7f99b414d8c0 msgr2=0x7f99b4064440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:11:39.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.957+0000 7f99b27fc700 1 --2- 172.21.15.88:0/759498365 >> v2:172.21.15.151:3300/0 conn(0x7f99b414d8c0 0x7f99b4064440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:39.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.957+0000 7f99b27fc700 1 -- 172.21.15.88:0/759498365 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99b40c72c0 con 0x7f99b4065360 2023-10-18T16:11:39.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.957+0000 7f99b27fc700 1 -- 172.21.15.88:0/759498365 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441715075 0 0) 0x7f99ac001470 con 0x7f99b4065360 2023-10-18T16:11:39.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.957+0000 7f99b27fc700 1 -- 172.21.15.88:0/759498365 >> v1:172.21.15.88:6789/0 conn(0x7f99b4065360 legacy=0x7f99b4153ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:39.972 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.957+0000 7f99b27fc700 1 --2- 172.21.15.88:0/759498365 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f999c0038e0 0x7f999c003d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:39.972 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.957+0000 7f99b27fc700 1 --2- 172.21.15.88:0/759498365 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f99b40a9050 0x7f999c004690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:39.972 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.957+0000 7f99b27fc700 1 --2- 172.21.15.88:0/759498365 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f999c004b80 0x7f999c005000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:39.972 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.957+0000 7f99b27fc700 1 -- 172.21.15.88:0/759498365 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b4153a40 con 0x7f999c004b80 2023-10-18T16:11:39.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.957+0000 7f99b27fc700 1 -- 172.21.15.88:0/759498365 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b4063e70 con 0x7f999c0038e0 2023-10-18T16:11:39.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.957+0000 7f99b27fc700 1 -- 172.21.15.88:0/759498365 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f999c00b8e0 con 0x7f99b40a9050 2023-10-18T16:11:39.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.957+0000 7f99b3fff700 1 --2- 172.21.15.88:0/759498365 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f999c004b80 0x7f999c005000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:39.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.957+0000 7f99b37fe700 1 --2- 172.21.15.88:0/759498365 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f999c0038e0 0x7f999c003d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:39.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.957+0000 7f99b2ffd700 1 --2- 172.21.15.88:0/759498365 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f99b40a9050 0x7f999c004690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:39.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.957+0000 7f99b3fff700 1 -- 172.21.15.88:0/759498365 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f99b40a9050 msgr2=0x7f999c004690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:11:39.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.957+0000 7f99b3fff700 1 --2- 172.21.15.88:0/759498365 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f99b40a9050 0x7f999c004690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:39.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.957+0000 7f99b3fff700 1 -- 172.21.15.88:0/759498365 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f999c0038e0 msgr2=0x7f999c003d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:11:39.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.957+0000 7f99b3fff700 1 --2- 172.21.15.88:0/759498365 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f999c0038e0 0x7f999c003d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:39.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.957+0000 7f99b3fff700 1 -- 172.21.15.88:0/759498365 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f999c0033f0 con 0x7f999c004b80 2023-10-18T16:11:39.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.957+0000 7f99b3fff700 1 --2- 172.21.15.88:0/759498365 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f999c004b80 0x7f999c005000 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f99ac00c210 tx=0x7f99ac001e90).ready entity=mon.0 client_cookie=9e7bc8fb27a3394f server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:11:39.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.957+0000 7f99b27fc700 1 -- 172.21.15.88:0/759498365 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99ac007a60 con 0x7f999c004b80 2023-10-18T16:11:39.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.957+0000 7f99b27fc700 1 -- 172.21.15.88:0/759498365 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99ac00cae0 con 0x7f999c004b80 2023-10-18T16:11:39.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.957+0000 7f99b27fc700 1 -- 172.21.15.88:0/759498365 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99ac00ce60 con 0x7f999c004b80 2023-10-18T16:11:39.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.958+0000 7f99b9cac700 1 -- 172.21.15.88:0/759498365 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f999c004b80 msgr2=0x7f999c005000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:39.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.958+0000 7f99b9cac700 1 --2- 172.21.15.88:0/759498365 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f999c004b80 0x7f999c005000 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f99ac00c210 tx=0x7f99ac001e90).stop 2023-10-18T16:11:39.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.958+0000 7f99b3fff700 1 -- 172.21.15.88:0/759498365 reap_dead start 2023-10-18T16:11:39.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.958+0000 7f99b9cac700 1 -- 172.21.15.88:0/759498365 shutdown_connections 2023-10-18T16:11:39.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.958+0000 7f99b9cac700 1 -- 172.21.15.88:0/759498365 >> 172.21.15.88:0/759498365 conn(0x7f99b406a810 msgr2=0x7f99b40c6a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:11:39.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.958+0000 7f99b9cac700 1 -- 172.21.15.88:0/759498365 shutdown_connections 2023-10-18T16:11:39.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.958+0000 7f99b9cac700 1 -- 172.21.15.88:0/759498365 wait complete. 2023-10-18T16:11:39.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.959+0000 7f99b9cac700 1 Processor -- start 2023-10-18T16:11:39.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.959+0000 7f99b9cac700 1 -- start start 2023-10-18T16:11:39.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.960+0000 7f99b9cac700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f999c004b80 0x7f99b40aabb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:39.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.960+0000 7f99b9cac700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f99b40bd5f0 0x7f99b40bca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:39.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.960+0000 7f99b9cac700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f99b40bcf50 0x7f99b40ab750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:39.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.960+0000 7f99b9cac700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b40ab1f0 con 0x7f99b40bd5f0 2023-10-18T16:11:39.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.960+0000 7f99b9cac700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b414fa10 con 0x7f99b40bcf50 2023-10-18T16:11:39.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.960+0000 7f99b9cac700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99b40c2380 con 0x7f999c004b80 2023-10-18T16:11:39.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.960+0000 7f99b2ffd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f99b40bd5f0 0x7f99b40bca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:39.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.960+0000 7f99b2ffd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f99b40bd5f0 0x7f99b40bca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:51936/0 (socket says 172.21.15.88:51936) 2023-10-18T16:11:39.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.960+0000 7f99b2ffd700 1 -- 172.21.15.88:0/2751388914 learned_addr learned my addr 172.21.15.88:0/2751388914 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:11:39.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.960+0000 7f99b3fff700 1 --2- 172.21.15.88:0/2751388914 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f99b40bcf50 0x7f99b40ab750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:39.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.960+0000 7f99b37fe700 1 --2- 172.21.15.88:0/2751388914 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f999c004b80 0x7f99b40aabb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:39.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.960+0000 7f99b2ffd700 1 -- 172.21.15.88:0/2751388914 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f999c004b80 msgr2=0x7f99b40aabb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:39.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.960+0000 7f99b2ffd700 1 --2- 172.21.15.88:0/2751388914 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f999c004b80 0x7f99b40aabb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:39.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.960+0000 7f99b2ffd700 1 -- 172.21.15.88:0/2751388914 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f99b40bcf50 msgr2=0x7f99b40ab750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:39.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.960+0000 7f99b2ffd700 1 --2- 172.21.15.88:0/2751388914 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f99b40bcf50 0x7f99b40ab750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:39.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.960+0000 7f99b2ffd700 1 -- 172.21.15.88:0/2751388914 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99b40a9bb0 con 0x7f99b40bd5f0 2023-10-18T16:11:39.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.960+0000 7f99b37fe700 1 --2- 172.21.15.88:0/2751388914 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f999c004b80 0x7f99b40aabb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:11:39.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.961+0000 7f99b2ffd700 1 --2- 172.21.15.88:0/2751388914 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f99b40bd5f0 0x7f99b40bca60 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f99a80119f0 tx=0x7f99a8009060).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:11:39.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.961+0000 7f999b7fe700 1 -- 172.21.15.88:0/2751388914 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a8011ee0 con 0x7f99b40bd5f0 2023-10-18T16:11:39.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.961+0000 7f999b7fe700 1 -- 172.21.15.88:0/2751388914 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99a800f8e0 con 0x7f99b40bd5f0 2023-10-18T16:11:39.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.961+0000 7f999b7fe700 1 -- 172.21.15.88:0/2751388914 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a800fca0 con 0x7f99b40bd5f0 2023-10-18T16:11:39.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.961+0000 7f99b9cac700 1 -- 172.21.15.88:0/2751388914 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99b40a9e90 con 0x7f99b40bd5f0 2023-10-18T16:11:39.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.961+0000 7f99b9cac700 1 -- 172.21.15.88:0/2751388914 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99b40aa450 con 0x7f99b40bd5f0 2023-10-18T16:11:39.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.962+0000 7f99b9cac700 1 -- 172.21.15.88:0/2751388914 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99b406c4e0 con 0x7f99b40bd5f0 2023-10-18T16:11:39.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.962+0000 7f999b7fe700 1 -- 172.21.15.88:0/2751388914 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f99a8014650 con 0x7f99b40bd5f0 2023-10-18T16:11:39.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.962+0000 7f999b7fe700 1 --2- 172.21.15.88:0/2751388914 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f99a005f150 0x7f99a0061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:39.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.962+0000 7f999b7fe700 1 -- 172.21.15.88:0/2751388914 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6725+0+0 (secure 0 0 0) 0x7f99a807afe0 con 0x7f99b40bd5f0 2023-10-18T16:11:39.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.963+0000 7f99b37fe700 1 --2- 172.21.15.88:0/2751388914 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f99a005f150 0x7f99a0061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:39.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.963+0000 7f99b37fe700 1 --2- 172.21.15.88:0/2751388914 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f99a005f150 0x7f99a0061610 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f99a40094a0 tx=0x7f99a400b040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:11:39.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:39.967+0000 7f999b7fe700 1 -- 172.21.15.88:0/2751388914 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f99a804c050 con 0x7f99b40bd5f0 2023-10-18T16:11:40.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:40.168+0000 7f99b9cac700 1 -- 172.21.15.88:0/2751388914 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f99b4065140 con 0x7f99b40bd5f0 2023-10-18T16:11:40.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:40.168+0000 7f999b7fe700 1 -- 172.21.15.88:0/2751388914 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7f99a804f670 con 0x7f99b40bd5f0 2023-10-18T16:11:40.172 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:11:40.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:40.170+0000 7f99b9cac700 1 -- 172.21.15.88:0/2751388914 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f99a005f150 msgr2=0x7f99a0061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:40.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:40.170+0000 7f99b9cac700 1 --2- 172.21.15.88:0/2751388914 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f99a005f150 0x7f99a0061610 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f99a40094a0 tx=0x7f99a400b040).stop 2023-10-18T16:11:40.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:40.170+0000 7f99b9cac700 1 -- 172.21.15.88:0/2751388914 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f99b40bd5f0 msgr2=0x7f99b40bca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:40.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:40.170+0000 7f99b9cac700 1 --2- 172.21.15.88:0/2751388914 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f99b40bd5f0 0x7f99b40bca60 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f99a80119f0 tx=0x7f99a8009060).stop 2023-10-18T16:11:40.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:40.170+0000 7f99b9cac700 1 -- 172.21.15.88:0/2751388914 shutdown_connections 2023-10-18T16:11:40.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:40.170+0000 7f99b9cac700 1 --2- 172.21.15.88:0/2751388914 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f99a005f150 0x7f99a0061610 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:40.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:40.170+0000 7f99b9cac700 1 --2- 172.21.15.88:0/2751388914 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f99b40bcf50 0x7f99b40ab750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:40.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:40.170+0000 7f99b9cac700 1 --2- 172.21.15.88:0/2751388914 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f999c004b80 0x7f99b40aabb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:40.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:40.170+0000 7f99b9cac700 1 --2- 172.21.15.88:0/2751388914 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f99b40bd5f0 0x7f99b40bca60 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:40.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:40.170+0000 7f99b9cac700 1 -- 172.21.15.88:0/2751388914 >> 172.21.15.88:0/2751388914 conn(0x7f99b406a810 msgr2=0x7f99b4153dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:11:40.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:40.170+0000 7f99b9cac700 1 -- 172.21.15.88:0/2751388914 shutdown_connections 2023-10-18T16:11:40.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:40.170+0000 7f99b9cac700 1 -- 172.21.15.88:0/2751388914 wait complete. 2023-10-18T16:11:40.186 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:11:42.109 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~126s 2023-10-18T16:11:42.110 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~139s 2023-10-18T16:11:42.110 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~172s 2023-10-18T16:11:42.111 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~152s 2023-10-18T16:11:42.111 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~233s 2023-10-18T16:11:42.111 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.f is failed for ~185s 2023-10-18T16:11:43.187 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:11:43.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.308+0000 7f4be6d8d700 1 Processor -- start 2023-10-18T16:11:43.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.308+0000 7f4be6d8d700 1 -- start start 2023-10-18T16:11:43.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.309+0000 7f4be6d8d700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4be00a9050 0x7f4be0153ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:43.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.309+0000 7f4be6d8d700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4be0064150 con 0x7f4be0065360 2023-10-18T16:11:43.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.309+0000 7f4be6d8d700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4be0153a40 con 0x7f4be014d8c0 2023-10-18T16:11:43.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.309+0000 7f4be6d8d700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4be0063e70 con 0x7f4be00a9050 2023-10-18T16:11:43.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.309+0000 7f4bdffff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4be00a9050 0x7f4be0153ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:43.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.309+0000 7f4be532a700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f4be0065360 0x7f4be015acb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:49494/0 (socket says 172.21.15.88:49494) 2023-10-18T16:11:43.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.309+0000 7f4bdffff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4be00a9050 0x7f4be0153ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:51944/0 (socket says 172.21.15.88:51944) 2023-10-18T16:11:43.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.309+0000 7f4be532a700 1 -- 172.21.15.88:0/253931558 learned_addr learned my addr 172.21.15.88:0/253931558 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:11:43.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.309+0000 7f4bdffff700 1 -- 172.21.15.88:0/253931558 >> v1:172.21.15.151:6789/0 conn(0x7f4be014d8c0 legacy=0x7f4be0064440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:11:43.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.309+0000 7f4bdffff700 1 -- 172.21.15.88:0/253931558 >> v1:172.21.15.88:6789/0 conn(0x7f4be0065360 legacy=0x7f4be015acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:43.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.309+0000 7f4bdffff700 1 -- 172.21.15.88:0/253931558 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4be00c72c0 con 0x7f4be00a9050 2023-10-18T16:11:43.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.309+0000 7f4bdf7fe700 1 -- 172.21.15.88:0/253931558 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4237735858 0 0) 0x7f4be0064150 con 0x7f4be0065360 2023-10-18T16:11:43.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.309+0000 7f4bdffff700 1 --2- 172.21.15.88:0/253931558 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4be00a9050 0x7f4be0153ed0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f4bcc009760 tx=0x7f4bcc009060).ready entity=mon.0 client_cookie=ac3e036805adf1ee server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:11:43.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.309+0000 7f4bdf7fe700 1 -- 172.21.15.88:0/253931558 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bcc0090d0 con 0x7f4be00a9050 2023-10-18T16:11:43.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.309+0000 7f4bdf7fe700 1 -- 172.21.15.88:0/253931558 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bcc00b520 con 0x7f4be00a9050 2023-10-18T16:11:43.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.309+0000 7f4bdf7fe700 1 -- 172.21.15.88:0/253931558 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bcc00b8a0 con 0x7f4be00a9050 2023-10-18T16:11:43.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.310+0000 7f4be6d8d700 1 -- 172.21.15.88:0/253931558 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4be00a9050 msgr2=0x7f4be0153ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:43.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.310+0000 7f4be6d8d700 1 --2- 172.21.15.88:0/253931558 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4be00a9050 0x7f4be0153ed0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f4bcc009760 tx=0x7f4bcc009060).stop 2023-10-18T16:11:43.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.310+0000 7f4be6d8d700 1 -- 172.21.15.88:0/253931558 shutdown_connections 2023-10-18T16:11:43.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.310+0000 7f4be6d8d700 1 --2- 172.21.15.88:0/253931558 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4be00a9050 0x7f4be0153ed0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:43.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.310+0000 7f4be6d8d700 1 -- 172.21.15.88:0/253931558 >> 172.21.15.88:0/253931558 conn(0x7f4be006a810 msgr2=0x7f4be00c6a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:11:43.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.310+0000 7f4be6d8d700 1 -- 172.21.15.88:0/253931558 shutdown_connections 2023-10-18T16:11:43.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.310+0000 7f4be6d8d700 1 -- 172.21.15.88:0/253931558 wait complete. 2023-10-18T16:11:43.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.311+0000 7f4be6d8d700 1 Processor -- start 2023-10-18T16:11:43.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.311+0000 7f4be6d8d700 1 -- start start 2023-10-18T16:11:43.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.312+0000 7f4be6d8d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4be0065360 0x7f4be00bf7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:43.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.312+0000 7f4be6d8d700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4be00a9050 0x7f4be00bfc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:43.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.312+0000 7f4be6d8d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4be014d8c0 0x7f4be00c0570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:43.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.312+0000 7f4be6d8d700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4be00c0180 con 0x7f4be014d8c0 2023-10-18T16:11:43.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.312+0000 7f4be6d8d700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4be00bc900 con 0x7f4be0065360 2023-10-18T16:11:43.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.312+0000 7f4be6d8d700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4be00bcbe0 con 0x7f4be00a9050 2023-10-18T16:11:43.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.312+0000 7f4be532a700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4be014d8c0 0x7f4be00c0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:43.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.312+0000 7f4be532a700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4be014d8c0 0x7f4be00c0570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:51960/0 (socket says 172.21.15.88:51960) 2023-10-18T16:11:43.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.312+0000 7f4be532a700 1 -- 172.21.15.88:0/1975792667 learned_addr learned my addr 172.21.15.88:0/1975792667 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:11:43.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.312+0000 7f4bdffff700 1 --2- 172.21.15.88:0/1975792667 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4be00a9050 0x7f4be00bfc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:43.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.312+0000 7f4be4b29700 1 --2- 172.21.15.88:0/1975792667 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4be0065360 0x7f4be00bf7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:43.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.313+0000 7f4be532a700 1 -- 172.21.15.88:0/1975792667 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4be00a9050 msgr2=0x7f4be00bfc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:43.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.313+0000 7f4be532a700 1 --2- 172.21.15.88:0/1975792667 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4be00a9050 0x7f4be00bfc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:43.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.313+0000 7f4be532a700 1 -- 172.21.15.88:0/1975792667 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4be0065360 msgr2=0x7f4be00bf7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:43.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.313+0000 7f4be532a700 1 --2- 172.21.15.88:0/1975792667 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4be0065360 0x7f4be00bf7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:43.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.313+0000 7f4be532a700 1 -- 172.21.15.88:0/1975792667 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bcc0092a0 con 0x7f4be014d8c0 2023-10-18T16:11:43.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.313+0000 7f4be4b29700 1 --2- 172.21.15.88:0/1975792667 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4be0065360 0x7f4be00bf7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:11:43.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.313+0000 7f4bdffff700 1 --2- 172.21.15.88:0/1975792667 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4be00a9050 0x7f4be00bfc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:11:43.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.313+0000 7f4be532a700 1 --2- 172.21.15.88:0/1975792667 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4be014d8c0 0x7f4be00c0570 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7f4bd800c790 tx=0x7f4bd8007f30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:11:43.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.313+0000 7f4bdcff9700 1 -- 172.21.15.88:0/1975792667 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd8005d30 con 0x7f4be014d8c0 2023-10-18T16:11:43.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.313+0000 7f4bdcff9700 1 -- 172.21.15.88:0/1975792667 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bd8007110 con 0x7f4be014d8c0 2023-10-18T16:11:43.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.313+0000 7f4be6d8d700 1 -- 172.21.15.88:0/1975792667 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4be00bcec0 con 0x7f4be014d8c0 2023-10-18T16:11:43.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.313+0000 7f4bdcff9700 1 -- 172.21.15.88:0/1975792667 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd80074d0 con 0x7f4be014d8c0 2023-10-18T16:11:43.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.313+0000 7f4be6d8d700 1 -- 172.21.15.88:0/1975792667 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4be00bd480 con 0x7f4be014d8c0 2023-10-18T16:11:43.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.314+0000 7f4be6d8d700 1 -- 172.21.15.88:0/1975792667 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4be006c4e0 con 0x7f4be014d8c0 2023-10-18T16:11:43.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.314+0000 7f4bdcff9700 1 -- 172.21.15.88:0/1975792667 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f4bd8007840 con 0x7f4be014d8c0 2023-10-18T16:11:43.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.314+0000 7f4bdcff9700 1 --2- 172.21.15.88:0/1975792667 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f4bc405f120 0x7f4bc40615e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:43.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.315+0000 7f4bdcff9700 1 -- 172.21.15.88:0/1975792667 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6725+0+0 (secure 0 0 0) 0x7f4bd807a4f0 con 0x7f4be014d8c0 2023-10-18T16:11:43.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.315+0000 7f4be4b29700 1 --2- 172.21.15.88:0/1975792667 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f4bc405f120 0x7f4bc40615e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:43.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.315+0000 7f4be4b29700 1 --2- 172.21.15.88:0/1975792667 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f4bc405f120 0x7f4bc40615e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f4bd400b8b0 tx=0x7f4bd4008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:11:43.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.319+0000 7f4bdcff9700 1 -- 172.21.15.88:0/1975792667 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f4bd804b560 con 0x7f4be014d8c0 2023-10-18T16:11:43.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.522+0000 7f4be6d8d700 1 -- 172.21.15.88:0/1975792667 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4be00c75a0 con 0x7f4be014d8c0 2023-10-18T16:11:43.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.523+0000 7f4bdcff9700 1 -- 172.21.15.88:0/1975792667 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7f4bd804eb80 con 0x7f4be014d8c0 2023-10-18T16:11:43.527 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:11:43.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.524+0000 7f4be6d8d700 1 -- 172.21.15.88:0/1975792667 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f4bc405f120 msgr2=0x7f4bc40615e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:43.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.524+0000 7f4be6d8d700 1 --2- 172.21.15.88:0/1975792667 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f4bc405f120 0x7f4bc40615e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f4bd400b8b0 tx=0x7f4bd4008040).stop 2023-10-18T16:11:43.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.524+0000 7f4be6d8d700 1 -- 172.21.15.88:0/1975792667 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4be014d8c0 msgr2=0x7f4be00c0570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:43.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.524+0000 7f4be6d8d700 1 --2- 172.21.15.88:0/1975792667 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4be014d8c0 0x7f4be00c0570 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7f4bd800c790 tx=0x7f4bd8007f30).stop 2023-10-18T16:11:43.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.524+0000 7f4be6d8d700 1 -- 172.21.15.88:0/1975792667 shutdown_connections 2023-10-18T16:11:43.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.524+0000 7f4be6d8d700 1 --2- 172.21.15.88:0/1975792667 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f4bc405f120 0x7f4bc40615e0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:43.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.524+0000 7f4be6d8d700 1 --2- 172.21.15.88:0/1975792667 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4be014d8c0 0x7f4be00c0570 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:43.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.524+0000 7f4be6d8d700 1 --2- 172.21.15.88:0/1975792667 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4be0065360 0x7f4be00bf7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:43.530 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.524+0000 7f4be6d8d700 1 --2- 172.21.15.88:0/1975792667 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4be00a9050 0x7f4be00bfc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:43.530 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.524+0000 7f4be6d8d700 1 -- 172.21.15.88:0/1975792667 >> 172.21.15.88:0/1975792667 conn(0x7f4be006a810 msgr2=0x7f4be00c19b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:11:43.530 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.524+0000 7f4be6d8d700 1 -- 172.21.15.88:0/1975792667 shutdown_connections 2023-10-18T16:11:43.530 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:43.524+0000 7f4be6d8d700 1 -- 172.21.15.88:0/1975792667 wait complete. 2023-10-18T16:11:43.542 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:11:46.544 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:11:46.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.665+0000 7f2219a67700 1 Processor -- start 2023-10-18T16:11:46.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.665+0000 7f2219a67700 1 -- start start 2023-10-18T16:11:46.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.666+0000 7f2219a67700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f22140bf230 0x7f22140a98c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:46.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.666+0000 7f2219a67700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2214063c70 con 0x7f2214153aa0 2023-10-18T16:11:46.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.666+0000 7f2219a67700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22140bf700 con 0x7f22140654b0 2023-10-18T16:11:46.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.666+0000 7f2219a67700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2214069680 con 0x7f22140bf230 2023-10-18T16:11:46.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.666+0000 7f2213fff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f22140bf230 0x7f22140a98c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:46.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.666+0000 7f2213fff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f22140bf230 0x7f22140a98c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:51976/0 (socket says 172.21.15.88:51976) 2023-10-18T16:11:46.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.666+0000 7f22137fe700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f2214153aa0 0x7f2214153e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:49498/0 (socket says 172.21.15.88:49498) 2023-10-18T16:11:46.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.666+0000 7f2213fff700 1 -- 172.21.15.88:0/2305238743 learned_addr learned my addr 172.21.15.88:0/2305238743 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:11:46.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.666+0000 7f2213fff700 1 -- 172.21.15.88:0/2305238743 >> v1:172.21.15.151:6789/0 conn(0x7f22140654b0 legacy=0x7f2214066b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:11:46.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.666+0000 7f2213fff700 1 -- 172.21.15.88:0/2305238743 >> v1:172.21.15.88:6789/0 conn(0x7f2214153aa0 legacy=0x7f2214153e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:46.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.666+0000 7f22127fc700 1 -- 172.21.15.88:0/2305238743 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 619440228 0 0) 0x7f2214063c70 con 0x7f2214153aa0 2023-10-18T16:11:46.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.666+0000 7f2213fff700 1 -- 172.21.15.88:0/2305238743 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22140690c0 con 0x7f22140bf230 2023-10-18T16:11:46.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.666+0000 7f2213fff700 1 --2- 172.21.15.88:0/2305238743 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f22140bf230 0x7f22140a98c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f220c009f20 tx=0x7f220c009840).ready entity=mon.0 client_cookie=477d4f3149c04717 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:11:46.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.666+0000 7f22127fc700 1 -- 172.21.15.88:0/2305238743 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f220c00e070 con 0x7f22140bf230 2023-10-18T16:11:46.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.667+0000 7f22127fc700 1 -- 172.21.15.88:0/2305238743 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f220c00bb10 con 0x7f22140bf230 2023-10-18T16:11:46.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.667+0000 7f22127fc700 1 -- 172.21.15.88:0/2305238743 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f220c00be90 con 0x7f22140bf230 2023-10-18T16:11:46.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.667+0000 7f2219a67700 1 -- 172.21.15.88:0/2305238743 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f22140bf230 msgr2=0x7f22140a98c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:46.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.667+0000 7f2219a67700 1 --2- 172.21.15.88:0/2305238743 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f22140bf230 0x7f22140a98c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f220c009f20 tx=0x7f220c009840).stop 2023-10-18T16:11:46.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.667+0000 7f2219a67700 1 -- 172.21.15.88:0/2305238743 shutdown_connections 2023-10-18T16:11:46.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.667+0000 7f2219a67700 1 --2- 172.21.15.88:0/2305238743 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f22140bf230 0x7f22140a98c0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:46.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.667+0000 7f2219a67700 1 -- 172.21.15.88:0/2305238743 >> 172.21.15.88:0/2305238743 conn(0x7f22140c4890 msgr2=0x7f22140c4ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:11:46.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.667+0000 7f2219a67700 1 -- 172.21.15.88:0/2305238743 shutdown_connections 2023-10-18T16:11:46.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.667+0000 7f2219a67700 1 -- 172.21.15.88:0/2305238743 wait complete. 2023-10-18T16:11:46.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.668+0000 7f2219a67700 1 Processor -- start 2023-10-18T16:11:46.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.668+0000 7f2219a67700 1 -- start start 2023-10-18T16:11:46.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.669+0000 7f2219a67700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f22140654b0 0x7f22140bd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:46.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.669+0000 7f2219a67700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f22140bf230 0x7f22140bc5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:46.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.669+0000 7f2219a67700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2214153aa0 0x7f2214155d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:46.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.669+0000 7f2219a67700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22141604c0 con 0x7f22140bf230 2023-10-18T16:11:46.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.669+0000 7f2219a67700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2214161000 con 0x7f2214153aa0 2023-10-18T16:11:46.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.669+0000 7f2219a67700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22140bde90 con 0x7f22140654b0 2023-10-18T16:11:46.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.669+0000 7f2212ffd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f22140bf230 0x7f22140bc5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:46.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.669+0000 7f2212ffd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f22140bf230 0x7f22140bc5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:51980/0 (socket says 172.21.15.88:51980) 2023-10-18T16:11:46.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.669+0000 7f2212ffd700 1 -- 172.21.15.88:0/1352437432 learned_addr learned my addr 172.21.15.88:0/1352437432 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:11:46.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.669+0000 7f2213fff700 1 --2- 172.21.15.88:0/1352437432 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2214153aa0 0x7f2214155d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:46.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.669+0000 7f2212ffd700 1 -- 172.21.15.88:0/1352437432 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f22140654b0 msgr2=0x7f22140bd8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:46.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.669+0000 7f22137fe700 1 --2- 172.21.15.88:0/1352437432 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f22140654b0 0x7f22140bd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:46.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.669+0000 7f2212ffd700 1 --2- 172.21.15.88:0/1352437432 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f22140654b0 0x7f22140bd8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:46.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.669+0000 7f2212ffd700 1 -- 172.21.15.88:0/1352437432 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2214153aa0 msgr2=0x7f2214155d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:46.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.669+0000 7f2212ffd700 1 --2- 172.21.15.88:0/1352437432 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2214153aa0 0x7f2214155d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:46.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.669+0000 7f2212ffd700 1 -- 172.21.15.88:0/1352437432 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f220c009ae0 con 0x7f22140bf230 2023-10-18T16:11:46.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.669+0000 7f2213fff700 1 --2- 172.21.15.88:0/1352437432 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2214153aa0 0x7f2214155d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-10-18T16:11:46.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.670+0000 7f2212ffd700 1 --2- 172.21.15.88:0/1352437432 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f22140bf230 0x7f22140bc5d0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f220800c440 tx=0x7f2208007de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:11:46.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.670+0000 7f21fb7fe700 1 -- 172.21.15.88:0/1352437432 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f220800f470 con 0x7f22140bf230 2023-10-18T16:11:46.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.670+0000 7f21fb7fe700 1 -- 172.21.15.88:0/1352437432 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f220800f5d0 con 0x7f22140bf230 2023-10-18T16:11:46.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.670+0000 7f2219a67700 1 -- 172.21.15.88:0/1352437432 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22140c33f0 con 0x7f22140bf230 2023-10-18T16:11:46.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.670+0000 7f21fb7fe700 1 -- 172.21.15.88:0/1352437432 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f220800f990 con 0x7f22140bf230 2023-10-18T16:11:46.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.670+0000 7f2219a67700 1 -- 172.21.15.88:0/1352437432 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22141606d0 con 0x7f22140bf230 2023-10-18T16:11:46.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.671+0000 7f2219a67700 1 -- 172.21.15.88:0/1352437432 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22140bcac0 con 0x7f22140bf230 2023-10-18T16:11:46.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.671+0000 7f21fb7fe700 1 -- 172.21.15.88:0/1352437432 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f2208012400 con 0x7f22140bf230 2023-10-18T16:11:46.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.671+0000 7f21fb7fe700 1 --2- 172.21.15.88:0/1352437432 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f220005f180 0x7f2200061640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:46.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.672+0000 7f21fb7fe700 1 -- 172.21.15.88:0/1352437432 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6725+0+0 (secure 0 0 0) 0x7f2208079970 con 0x7f22140bf230 2023-10-18T16:11:46.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.672+0000 7f22137fe700 1 --2- 172.21.15.88:0/1352437432 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f220005f180 0x7f2200061640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:46.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.672+0000 7f22137fe700 1 --2- 172.21.15.88:0/1352437432 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f220005f180 0x7f2200061640 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f22040015d0 tx=0x7f2204009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:11:46.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.676+0000 7f21fb7fe700 1 -- 172.21.15.88:0/1352437432 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f2208048ac0 con 0x7f22140bf230 2023-10-18T16:11:46.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.878+0000 7f2219a67700 1 -- 172.21.15.88:0/1352437432 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f22140c39b0 con 0x7f22140bf230 2023-10-18T16:11:46.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.878+0000 7f21fb7fe700 1 -- 172.21.15.88:0/1352437432 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7f220804aa30 con 0x7f22140bf230 2023-10-18T16:11:46.882 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:11:46.884 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.880+0000 7f2219a67700 1 -- 172.21.15.88:0/1352437432 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f220005f180 msgr2=0x7f2200061640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:46.884 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.880+0000 7f2219a67700 1 --2- 172.21.15.88:0/1352437432 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f220005f180 0x7f2200061640 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f22040015d0 tx=0x7f2204009040).stop 2023-10-18T16:11:46.885 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.880+0000 7f2219a67700 1 -- 172.21.15.88:0/1352437432 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f22140bf230 msgr2=0x7f22140bc5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:46.885 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.880+0000 7f2219a67700 1 --2- 172.21.15.88:0/1352437432 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f22140bf230 0x7f22140bc5d0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f220800c440 tx=0x7f2208007de0).stop 2023-10-18T16:11:46.885 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.880+0000 7f2219a67700 1 -- 172.21.15.88:0/1352437432 shutdown_connections 2023-10-18T16:11:46.885 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.880+0000 7f2219a67700 1 --2- 172.21.15.88:0/1352437432 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f220005f180 0x7f2200061640 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:46.885 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.880+0000 7f2219a67700 1 --2- 172.21.15.88:0/1352437432 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2214153aa0 0x7f2214155d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:46.886 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.880+0000 7f2219a67700 1 --2- 172.21.15.88:0/1352437432 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f22140654b0 0x7f22140bd8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:46.886 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.880+0000 7f2219a67700 1 --2- 172.21.15.88:0/1352437432 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f22140bf230 0x7f22140bc5d0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:46.886 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.880+0000 7f2219a67700 1 -- 172.21.15.88:0/1352437432 >> 172.21.15.88:0/1352437432 conn(0x7f22140c4890 msgr2=0x7f22140c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:11:46.886 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.880+0000 7f2219a67700 1 -- 172.21.15.88:0/1352437432 shutdown_connections 2023-10-18T16:11:46.886 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:46.880+0000 7f2219a67700 1 -- 172.21.15.88:0/1352437432 wait complete. 2023-10-18T16:11:46.895 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:11:48.421 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~133s 2023-10-18T16:11:48.421 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~145s 2023-10-18T16:11:48.421 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~178s 2023-10-18T16:11:48.422 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~158s 2023-10-18T16:11:48.422 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~239s 2023-10-18T16:11:48.422 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.f is failed for ~192s 2023-10-18T16:11:49.897 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:11:50.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.018+0000 7ff1519e6700 1 Processor -- start 2023-10-18T16:11:50.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.018+0000 7ff1519e6700 1 -- start start 2023-10-18T16:11:50.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.019+0000 7ff1519e6700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff14c0bcb70 0x7ff14c0bcf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:50.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.019+0000 7ff1519e6700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff14c063f70 con 0x7ff14c0bb940 2023-10-18T16:11:50.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.019+0000 7ff1519e6700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff14c062dc0 con 0x7ff14c150bf0 2023-10-18T16:11:50.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.019+0000 7ff1519e6700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff14c151030 con 0x7ff14c0bcb70 2023-10-18T16:11:50.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.019+0000 7ff14affd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff14c0bcb70 0x7ff14c0bcf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:50.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.019+0000 7ff14affd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff14c0bcb70 0x7ff14c0bcf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:53032/0 (socket says 172.21.15.88:53032) 2023-10-18T16:11:50.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.019+0000 7ff14b7fe700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7ff14c0bb940 0x7ff14c0c1380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:42314/0 (socket says 172.21.15.88:42314) 2023-10-18T16:11:50.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.019+0000 7ff14affd700 1 -- 172.21.15.88:0/2970631504 learned_addr learned my addr 172.21.15.88:0/2970631504 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:11:50.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.019+0000 7ff14affd700 1 -- 172.21.15.88:0/2970631504 >> v1:172.21.15.151:6789/0 conn(0x7ff14c150bf0 legacy=0x7ff14c14d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:11:50.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.020+0000 7ff14affd700 1 -- 172.21.15.88:0/2970631504 >> v1:172.21.15.88:6789/0 conn(0x7ff14c0bb940 legacy=0x7ff14c0c1380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:50.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.020+0000 7ff14affd700 1 -- 172.21.15.88:0/2970631504 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff14c063080 con 0x7ff14c0bcb70 2023-10-18T16:11:50.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.020+0000 7ff14a7fc700 1 -- 172.21.15.88:0/2970631504 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2848963995 0 0) 0x7ff14c063f70 con 0x7ff14c0bb940 2023-10-18T16:11:50.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.020+0000 7ff14affd700 1 --2- 172.21.15.88:0/2970631504 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff14c0bcb70 0x7ff14c0bcf50 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7ff13c006fd0 tx=0x7ff13c0090f0).ready entity=mon.0 client_cookie=c3ed63da9cbb061f server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:11:50.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.020+0000 7ff14a7fc700 1 -- 172.21.15.88:0/2970631504 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff13c00b600 con 0x7ff14c0bcb70 2023-10-18T16:11:50.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.020+0000 7ff14a7fc700 1 -- 172.21.15.88:0/2970631504 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff13c00b760 con 0x7ff14c0bcb70 2023-10-18T16:11:50.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.020+0000 7ff14a7fc700 1 -- 172.21.15.88:0/2970631504 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff13c00bae0 con 0x7ff14c0bcb70 2023-10-18T16:11:50.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.020+0000 7ff1519e6700 1 -- 172.21.15.88:0/2970631504 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff14c0bcb70 msgr2=0x7ff14c0bcf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:50.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.020+0000 7ff1519e6700 1 --2- 172.21.15.88:0/2970631504 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff14c0bcb70 0x7ff14c0bcf50 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7ff13c006fd0 tx=0x7ff13c0090f0).stop 2023-10-18T16:11:50.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.020+0000 7ff1519e6700 1 -- 172.21.15.88:0/2970631504 shutdown_connections 2023-10-18T16:11:50.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.020+0000 7ff1519e6700 1 --2- 172.21.15.88:0/2970631504 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff14c0bcb70 0x7ff14c0bcf50 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:50.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.020+0000 7ff1519e6700 1 -- 172.21.15.88:0/2970631504 >> 172.21.15.88:0/2970631504 conn(0x7ff14c0c69e0 msgr2=0x7ff14c0c6df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:11:50.029 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.020+0000 7ff1519e6700 1 -- 172.21.15.88:0/2970631504 shutdown_connections 2023-10-18T16:11:50.029 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.020+0000 7ff1519e6700 1 -- 172.21.15.88:0/2970631504 wait complete. 2023-10-18T16:11:50.032 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.022+0000 7ff1519e6700 1 Processor -- start 2023-10-18T16:11:50.032 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.022+0000 7ff1519e6700 1 -- start start 2023-10-18T16:11:50.032 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.022+0000 7ff1519e6700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff14c0bb940 0x7ff14c14ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:50.032 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.022+0000 7ff1519e6700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff14c0bcb70 0x7ff14c163620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:50.032 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.022+0000 7ff1519e6700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff14c150bf0 0x7ff14c163b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:50.033 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.022+0000 7ff1519e6700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff14c0aa290 con 0x7ff14c150bf0 2023-10-18T16:11:50.033 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.022+0000 7ff1519e6700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff14c0aa070 con 0x7ff14c0bb940 2023-10-18T16:11:50.033 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.022+0000 7ff1519e6700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff14c1600d0 con 0x7ff14c0bcb70 2023-10-18T16:11:50.033 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.022+0000 7ff14b7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff14c150bf0 0x7ff14c163b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:50.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.022+0000 7ff14b7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff14c150bf0 0x7ff14c163b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:53042/0 (socket says 172.21.15.88:53042) 2023-10-18T16:11:50.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.022+0000 7ff14b7fe700 1 -- 172.21.15.88:0/3207265588 learned_addr learned my addr 172.21.15.88:0/3207265588 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:11:50.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.023+0000 7ff14affd700 1 --2- 172.21.15.88:0/3207265588 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff14c0bb940 0x7ff14c14ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:50.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.023+0000 7ff1427fc700 1 --2- 172.21.15.88:0/3207265588 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff14c0bcb70 0x7ff14c163620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:50.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.023+0000 7ff14b7fe700 1 -- 172.21.15.88:0/3207265588 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff14c0bcb70 msgr2=0x7ff14c163620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:50.035 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.023+0000 7ff14b7fe700 1 --2- 172.21.15.88:0/3207265588 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff14c0bcb70 0x7ff14c163620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:50.035 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.023+0000 7ff14b7fe700 1 -- 172.21.15.88:0/3207265588 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff14c0bb940 msgr2=0x7ff14c14ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:50.035 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.023+0000 7ff14b7fe700 1 --2- 172.21.15.88:0/3207265588 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff14c0bb940 0x7ff14c14ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:50.035 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.023+0000 7ff14b7fe700 1 -- 172.21.15.88:0/3207265588 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff13c009370 con 0x7ff14c150bf0 2023-10-18T16:11:50.035 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.023+0000 7ff1427fc700 1 --2- 172.21.15.88:0/3207265588 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff14c0bcb70 0x7ff14c163620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-10-18T16:11:50.036 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.023+0000 7ff14b7fe700 1 --2- 172.21.15.88:0/3207265588 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff14c150bf0 0x7ff14c163b10 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7ff14400c7e0 tx=0x7ff144007f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:11:50.036 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.023+0000 7ff1437fe700 1 -- 172.21.15.88:0/3207265588 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff144005c70 con 0x7ff14c150bf0 2023-10-18T16:11:50.036 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.023+0000 7ff1437fe700 1 -- 172.21.15.88:0/3207265588 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff144005dd0 con 0x7ff14c150bf0 2023-10-18T16:11:50.036 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.023+0000 7ff1519e6700 1 -- 172.21.15.88:0/3207265588 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff14c0a9d90 con 0x7ff14c150bf0 2023-10-18T16:11:50.036 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.023+0000 7ff1437fe700 1 -- 172.21.15.88:0/3207265588 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff144007320 con 0x7ff14c150bf0 2023-10-18T16:11:50.037 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.023+0000 7ff1519e6700 1 -- 172.21.15.88:0/3207265588 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff14c1605c0 con 0x7ff14c150bf0 2023-10-18T16:11:50.037 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.024+0000 7ff1519e6700 1 -- 172.21.15.88:0/3207265588 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff14c06c4e0 con 0x7ff14c150bf0 2023-10-18T16:11:50.037 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.024+0000 7ff1437fe700 1 -- 172.21.15.88:0/3207265588 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7ff144007710 con 0x7ff14c150bf0 2023-10-18T16:11:50.037 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.025+0000 7ff1437fe700 1 --2- 172.21.15.88:0/3207265588 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7ff12c05f150 0x7ff12c061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:50.037 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.025+0000 7ff1437fe700 1 -- 172.21.15.88:0/3207265588 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6725+0+0 (secure 0 0 0) 0x7ff14407a5a0 con 0x7ff14c150bf0 2023-10-18T16:11:50.038 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.025+0000 7ff14affd700 1 --2- 172.21.15.88:0/3207265588 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7ff12c05f150 0x7ff12c061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:50.038 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.025+0000 7ff14affd700 1 --2- 172.21.15.88:0/3207265588 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7ff12c05f150 0x7ff12c061610 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7ff13c009130 tx=0x7ff13c017000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:11:50.039 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.029+0000 7ff1437fe700 1 -- 172.21.15.88:0/3207265588 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7ff14404b610 con 0x7ff14c150bf0 2023-10-18T16:11:50.232 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.229+0000 7ff1519e6700 1 -- 172.21.15.88:0/3207265588 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff14c063c00 con 0x7ff14c150bf0 2023-10-18T16:11:50.233 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.229+0000 7ff1437fe700 1 -- 172.21.15.88:0/3207265588 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7ff14404ec30 con 0x7ff14c150bf0 2023-10-18T16:11:50.233 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:11:50.234 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.231+0000 7ff1519e6700 1 -- 172.21.15.88:0/3207265588 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7ff12c05f150 msgr2=0x7ff12c061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:50.235 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.231+0000 7ff1519e6700 1 --2- 172.21.15.88:0/3207265588 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7ff12c05f150 0x7ff12c061610 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7ff13c009130 tx=0x7ff13c017000).stop 2023-10-18T16:11:50.235 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.231+0000 7ff1519e6700 1 -- 172.21.15.88:0/3207265588 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff14c150bf0 msgr2=0x7ff14c163b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:50.235 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.231+0000 7ff1519e6700 1 --2- 172.21.15.88:0/3207265588 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff14c150bf0 0x7ff14c163b10 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7ff14400c7e0 tx=0x7ff144007f60).stop 2023-10-18T16:11:50.235 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.231+0000 7ff1519e6700 1 -- 172.21.15.88:0/3207265588 shutdown_connections 2023-10-18T16:11:50.235 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.231+0000 7ff1519e6700 1 --2- 172.21.15.88:0/3207265588 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7ff12c05f150 0x7ff12c061610 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:50.236 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.231+0000 7ff1519e6700 1 --2- 172.21.15.88:0/3207265588 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff14c150bf0 0x7ff14c163b10 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:50.236 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.231+0000 7ff1519e6700 1 --2- 172.21.15.88:0/3207265588 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff14c0bb940 0x7ff14c14ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:50.236 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.231+0000 7ff1519e6700 1 --2- 172.21.15.88:0/3207265588 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff14c0bcb70 0x7ff14c163620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:50.236 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.231+0000 7ff1519e6700 1 -- 172.21.15.88:0/3207265588 >> 172.21.15.88:0/3207265588 conn(0x7ff14c0c69e0 msgr2=0x7ff14c1472e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:11:50.236 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.231+0000 7ff1519e6700 1 -- 172.21.15.88:0/3207265588 shutdown_connections 2023-10-18T16:11:50.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:50.231+0000 7ff1519e6700 1 -- 172.21.15.88:0/3207265588 wait complete. 2023-10-18T16:11:50.246 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:11:53.247 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:11:53.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.368+0000 7f69894a5700 1 Processor -- start 2023-10-18T16:11:53.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.368+0000 7f69894a5700 1 -- start start 2023-10-18T16:11:53.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.368+0000 7f69894a5700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f698414a6c0 0x7f6984067290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:53.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.368+0000 7f69894a5700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f69840bd880 0x7f6984069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:53.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.368+0000 7f69894a5700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69840c4390 con 0x7f698406a090 2023-10-18T16:11:53.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.368+0000 7f69894a5700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f698406a490 con 0x7f698414a6c0 2023-10-18T16:11:53.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.368+0000 7f69894a5700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f698414aaa0 con 0x7f69840bd880 2023-10-18T16:11:53.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.369+0000 7f6982ffd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f698414a6c0 0x7f6984067290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:53.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.369+0000 7f6982ffd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f698414a6c0 0x7f6984067290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:53056/0 (socket says 172.21.15.88:53056) 2023-10-18T16:11:53.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.369+0000 7f6982ffd700 1 -- 172.21.15.88:0/3743845665 learned_addr learned my addr 172.21.15.88:0/3743845665 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:11:53.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.369+0000 7f69837fe700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f698406a090 0x7f69840bc2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:42322/0 (socket says 172.21.15.88:42322) 2023-10-18T16:11:53.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.369+0000 7f697a7fc700 1 --2- 172.21.15.88:0/3743845665 >> v2:172.21.15.151:3300/0 conn(0x7f69840bd880 0x7f6984069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:53.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.369+0000 7f6982ffd700 1 -- 172.21.15.88:0/3743845665 >> v2:172.21.15.151:3300/0 conn(0x7f69840bd880 msgr2=0x7f6984069ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:11:53.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.369+0000 7f6982ffd700 1 --2- 172.21.15.88:0/3743845665 >> v2:172.21.15.151:3300/0 conn(0x7f69840bd880 0x7f6984069ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:53.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.369+0000 7f6982ffd700 1 -- 172.21.15.88:0/3743845665 >> v1:172.21.15.88:6789/0 conn(0x7f698406a090 legacy=0x7f69840bc2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:53.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.369+0000 7f6982ffd700 1 -- 172.21.15.88:0/3743845665 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6984150570 con 0x7f698414a6c0 2023-10-18T16:11:53.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.369+0000 7f69827fc700 1 -- 172.21.15.88:0/3743845665 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 911650576 0 0) 0x7f69840c4390 con 0x7f698406a090 2023-10-18T16:11:53.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.369+0000 7f697a7fc700 1 --2- 172.21.15.88:0/3743845665 >> v2:172.21.15.151:3300/0 conn(0x7f69840bd880 0x7f6984069ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-10-18T16:11:53.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.369+0000 7f6982ffd700 1 --2- 172.21.15.88:0/3743845665 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f698414a6c0 0x7f6984067290 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7f698406aaf0 tx=0x7f69740090f0).ready entity=mon.0 client_cookie=552bacd1f810c8aa server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:11:53.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.369+0000 7f69827fc700 1 -- 172.21.15.88:0/3743845665 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f697400b600 con 0x7f698414a6c0 2023-10-18T16:11:53.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.369+0000 7f69827fc700 1 -- 172.21.15.88:0/3743845665 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f697400b760 con 0x7f698414a6c0 2023-10-18T16:11:53.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.369+0000 7f69827fc700 1 -- 172.21.15.88:0/3743845665 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f697400bb00 con 0x7f698414a6c0 2023-10-18T16:11:53.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.370+0000 7f69894a5700 1 -- 172.21.15.88:0/3743845665 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f698414a6c0 msgr2=0x7f6984067290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:53.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.370+0000 7f69894a5700 1 --2- 172.21.15.88:0/3743845665 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f698414a6c0 0x7f6984067290 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7f698406aaf0 tx=0x7f69740090f0).stop 2023-10-18T16:11:53.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.370+0000 7f69894a5700 1 -- 172.21.15.88:0/3743845665 shutdown_connections 2023-10-18T16:11:53.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.370+0000 7f69894a5700 1 --2- 172.21.15.88:0/3743845665 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f698414a6c0 0x7f6984067290 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:53.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.370+0000 7f69894a5700 1 --2- 172.21.15.88:0/3743845665 >> v2:172.21.15.151:3300/0 conn(0x7f69840bd880 0x7f6984069ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:53.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.370+0000 7f69894a5700 1 -- 172.21.15.88:0/3743845665 >> 172.21.15.88:0/3743845665 conn(0x7f6984065b10 msgr2=0x7f6984065f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:11:53.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.370+0000 7f69894a5700 1 -- 172.21.15.88:0/3743845665 shutdown_connections 2023-10-18T16:11:53.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.370+0000 7f69894a5700 1 -- 172.21.15.88:0/3743845665 wait complete. 2023-10-18T16:11:53.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.371+0000 7f69894a5700 1 Processor -- start 2023-10-18T16:11:53.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.371+0000 7f69894a5700 1 -- start start 2023-10-18T16:11:53.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.372+0000 7f69894a5700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f698406a090 0x7f69840afeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:53.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.372+0000 7f69894a5700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69840bd880 0x7f6984145fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:53.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.372+0000 7f69894a5700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f698414a6c0 0x7f69841464a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:53.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.372+0000 7f69894a5700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69840b2590 con 0x7f69840bd880 2023-10-18T16:11:53.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.372+0000 7f69894a5700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f698414c1e0 con 0x7f698406a090 2023-10-18T16:11:53.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.372+0000 7f69894a5700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f698414b900 con 0x7f698414a6c0 2023-10-18T16:11:53.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.372+0000 7f697a7fc700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69840bd880 0x7f6984145fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:53.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.372+0000 7f697a7fc700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69840bd880 0x7f6984145fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:53072/0 (socket says 172.21.15.88:53072) 2023-10-18T16:11:53.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.372+0000 7f697a7fc700 1 -- 172.21.15.88:0/3568186196 learned_addr learned my addr 172.21.15.88:0/3568186196 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:11:53.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.372+0000 7f6982ffd700 1 --2- 172.21.15.88:0/3568186196 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f698406a090 0x7f69840afeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:53.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.372+0000 7f69837fe700 1 --2- 172.21.15.88:0/3568186196 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f698414a6c0 0x7f69841464a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:53.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.372+0000 7f697a7fc700 1 -- 172.21.15.88:0/3568186196 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f698414a6c0 msgr2=0x7f69841464a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:53.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.372+0000 7f697a7fc700 1 --2- 172.21.15.88:0/3568186196 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f698414a6c0 0x7f69841464a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:53.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.372+0000 7f697a7fc700 1 -- 172.21.15.88:0/3568186196 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f698406a090 msgr2=0x7f69840afeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:53.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.372+0000 7f697a7fc700 1 --2- 172.21.15.88:0/3568186196 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f698406a090 0x7f69840afeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:53.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.372+0000 7f697a7fc700 1 -- 172.21.15.88:0/3568186196 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6974009370 con 0x7f69840bd880 2023-10-18T16:11:53.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.372+0000 7f697a7fc700 1 --2- 172.21.15.88:0/3568186196 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69840bd880 0x7f6984145fb0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7f696c00c8b0 tx=0x7f696c008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:11:53.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.373+0000 7f697b7fe700 1 -- 172.21.15.88:0/3568186196 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f696c010580 con 0x7f69840bd880 2023-10-18T16:11:53.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.373+0000 7f697b7fe700 1 -- 172.21.15.88:0/3568186196 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f696c0106e0 con 0x7f69840bd880 2023-10-18T16:11:53.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.373+0000 7f69894a5700 1 -- 172.21.15.88:0/3568186196 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f698414d6f0 con 0x7f69840bd880 2023-10-18T16:11:53.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.373+0000 7f697b7fe700 1 -- 172.21.15.88:0/3568186196 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f696c010a60 con 0x7f69840bd880 2023-10-18T16:11:53.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.373+0000 7f69894a5700 1 -- 172.21.15.88:0/3568186196 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f698414bb90 con 0x7f69840bd880 2023-10-18T16:11:53.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.374+0000 7f69894a5700 1 -- 172.21.15.88:0/3568186196 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6984068430 con 0x7f69840bd880 2023-10-18T16:11:53.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.374+0000 7f697b7fe700 1 -- 172.21.15.88:0/3568186196 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f696c01b460 con 0x7f69840bd880 2023-10-18T16:11:53.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.374+0000 7f697b7fe700 1 --2- 172.21.15.88:0/3568186196 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f696405f150 0x7f6964061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:53.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.374+0000 7f697b7fe700 1 -- 172.21.15.88:0/3568186196 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6725+0+0 (secure 0 0 0) 0x7f696c07a2e0 con 0x7f69840bd880 2023-10-18T16:11:53.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.375+0000 7f6982ffd700 1 --2- 172.21.15.88:0/3568186196 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f696405f150 0x7f6964061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:53.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.375+0000 7f6982ffd700 1 --2- 172.21.15.88:0/3568186196 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f696405f150 0x7f6964061610 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f6974009fd0 tx=0x7f6974017000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:11:53.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.378+0000 7f697b7fe700 1 -- 172.21.15.88:0/3568186196 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f696c04b2f0 con 0x7f69840bd880 2023-10-18T16:11:53.586 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.582+0000 7f69894a5700 1 -- 172.21.15.88:0/3568186196 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f698406c4e0 con 0x7f69840bd880 2023-10-18T16:11:53.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.582+0000 7f697b7fe700 1 -- 172.21.15.88:0/3568186196 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7f696c04e910 con 0x7f69840bd880 2023-10-18T16:11:53.587 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:11:53.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.584+0000 7f69894a5700 1 -- 172.21.15.88:0/3568186196 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f696405f150 msgr2=0x7f6964061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:53.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.584+0000 7f69894a5700 1 --2- 172.21.15.88:0/3568186196 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f696405f150 0x7f6964061610 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f6974009fd0 tx=0x7f6974017000).stop 2023-10-18T16:11:53.589 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.584+0000 7f69894a5700 1 -- 172.21.15.88:0/3568186196 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69840bd880 msgr2=0x7f6984145fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:53.589 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.584+0000 7f69894a5700 1 --2- 172.21.15.88:0/3568186196 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69840bd880 0x7f6984145fb0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7f696c00c8b0 tx=0x7f696c008040).stop 2023-10-18T16:11:53.589 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.584+0000 7f69894a5700 1 -- 172.21.15.88:0/3568186196 shutdown_connections 2023-10-18T16:11:53.589 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.584+0000 7f69894a5700 1 --2- 172.21.15.88:0/3568186196 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f696405f150 0x7f6964061610 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:53.589 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.584+0000 7f69894a5700 1 --2- 172.21.15.88:0/3568186196 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f698414a6c0 0x7f69841464a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:53.589 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.584+0000 7f69894a5700 1 --2- 172.21.15.88:0/3568186196 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f698406a090 0x7f69840afeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:53.590 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.584+0000 7f69894a5700 1 --2- 172.21.15.88:0/3568186196 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69840bd880 0x7f6984145fb0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:53.590 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.584+0000 7f69894a5700 1 -- 172.21.15.88:0/3568186196 >> 172.21.15.88:0/3568186196 conn(0x7f6984065b10 msgr2=0x7f69840bfe80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:11:53.590 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.584+0000 7f69894a5700 1 -- 172.21.15.88:0/3568186196 shutdown_connections 2023-10-18T16:11:53.590 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:53.585+0000 7f69894a5700 1 -- 172.21.15.88:0/3568186196 wait complete. 2023-10-18T16:11:53.600 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:11:54.732 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~139s 2023-10-18T16:11:54.732 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~152s 2023-10-18T16:11:54.732 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~185s 2023-10-18T16:11:54.732 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~165s 2023-10-18T16:11:54.733 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~246s 2023-10-18T16:11:54.733 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.f is failed for ~198s 2023-10-18T16:11:56.601 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:11:56.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.723+0000 7f23c6a30700 1 Processor -- start 2023-10-18T16:11:56.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.723+0000 7f23c6a30700 1 -- start start 2023-10-18T16:11:56.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.724+0000 7f23c6a30700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f23c01533f0 0x7f23c0151ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:56.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.724+0000 7f23c6a30700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f23c01523d0 0x7f23c00c36c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:56.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.724+0000 7f23c6a30700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23c00a9670 con 0x7f23c0153010 2023-10-18T16:11:56.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.724+0000 7f23c6a30700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23c014df50 con 0x7f23c01523d0 2023-10-18T16:11:56.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.724+0000 7f23c6a30700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23c0151a50 con 0x7f23c01533f0 2023-10-18T16:11:56.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.724+0000 7f23c4fcd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f23c01523d0 0x7f23c00c36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:56.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.724+0000 7f23c4fcd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f23c01523d0 0x7f23c00c36c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:53088/0 (socket says 172.21.15.88:53088) 2023-10-18T16:11:56.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.724+0000 7f23c4fcd700 1 -- 172.21.15.88:0/3599685230 learned_addr learned my addr 172.21.15.88:0/3599685230 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:11:56.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.724+0000 7f23bf7fe700 1 --2- 172.21.15.88:0/3599685230 >> v2:172.21.15.151:3300/0 conn(0x7f23c01533f0 0x7f23c0151ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:56.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.725+0000 7f23c4fcd700 1 -- 172.21.15.88:0/3599685230 >> v2:172.21.15.151:3300/0 conn(0x7f23c01533f0 msgr2=0x7f23c0151ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:11:56.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.725+0000 7f23c4fcd700 1 --2- 172.21.15.88:0/3599685230 >> v2:172.21.15.151:3300/0 conn(0x7f23c01533f0 0x7f23c0151ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:56.732 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.725+0000 7f23c4fcd700 1 -- 172.21.15.88:0/3599685230 >> v1:172.21.15.151:6789/0 conn(0x7f23c0153010 legacy=0x7f23c0064d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:56.732 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.725+0000 7f23c4fcd700 1 -- 172.21.15.88:0/3599685230 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23c00a8af0 con 0x7f23c01523d0 2023-10-18T16:11:56.732 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.725+0000 7f23c4fcd700 1 --2- 172.21.15.88:0/3599685230 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f23c01523d0 0x7f23c00c36c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7f23c005ea90 tx=0x7f23b8009840).ready entity=mon.0 client_cookie=90ac5d6705a114b7 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:11:56.732 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.725+0000 7f23beffd700 1 -- 172.21.15.88:0/3599685230 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b800e070 con 0x7f23c01523d0 2023-10-18T16:11:56.732 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.725+0000 7f23beffd700 1 -- 172.21.15.88:0/3599685230 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23b800ba50 con 0x7f23c01523d0 2023-10-18T16:11:56.732 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.725+0000 7f23beffd700 1 -- 172.21.15.88:0/3599685230 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b800bdd0 con 0x7f23c01523d0 2023-10-18T16:11:56.733 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.725+0000 7f23c6a30700 1 -- 172.21.15.88:0/3599685230 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f23c01523d0 msgr2=0x7f23c00c36c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:56.733 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.725+0000 7f23c6a30700 1 --2- 172.21.15.88:0/3599685230 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f23c01523d0 0x7f23c00c36c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7f23c005ea90 tx=0x7f23b8009840).stop 2023-10-18T16:11:56.733 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.725+0000 7f23c6a30700 1 -- 172.21.15.88:0/3599685230 shutdown_connections 2023-10-18T16:11:56.733 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.725+0000 7f23c6a30700 1 --2- 172.21.15.88:0/3599685230 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f23c01523d0 0x7f23c00c36c0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:56.733 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.725+0000 7f23c6a30700 1 --2- 172.21.15.88:0/3599685230 >> v2:172.21.15.151:3300/0 conn(0x7f23c01533f0 0x7f23c0151ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:56.734 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.725+0000 7f23c6a30700 1 -- 172.21.15.88:0/3599685230 >> 172.21.15.88:0/3599685230 conn(0x7f23c0069d80 msgr2=0x7f23c006a190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:11:56.734 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.725+0000 7f23c6a30700 1 -- 172.21.15.88:0/3599685230 shutdown_connections 2023-10-18T16:11:56.734 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.725+0000 7f23c6a30700 1 -- 172.21.15.88:0/3599685230 wait complete. 2023-10-18T16:11:56.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.726+0000 7f23c6a30700 1 Processor -- start 2023-10-18T16:11:56.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.727+0000 7f23c6a30700 1 -- start start 2023-10-18T16:11:56.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.728+0000 7f23c6a30700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f23c01523d0 0x7f23c014c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:56.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.728+0000 7f23c6a30700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f23c0153010 0x7f23c0161dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:56.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.728+0000 7f23c6a30700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f23c01533f0 0x7f23c01622b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:56.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.728+0000 7f23c6a30700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23c00a2970 con 0x7f23c0153010 2023-10-18T16:11:56.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.728+0000 7f23c6a30700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23c0156880 con 0x7f23c01523d0 2023-10-18T16:11:56.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.728+0000 7f23c6a30700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23c0161930 con 0x7f23c01533f0 2023-10-18T16:11:56.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.728+0000 7f23bf7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f23c0153010 0x7f23c0161dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:56.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.728+0000 7f23bf7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f23c0153010 0x7f23c0161dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:53102/0 (socket says 172.21.15.88:53102) 2023-10-18T16:11:56.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.728+0000 7f23bf7fe700 1 -- 172.21.15.88:0/4278712991 learned_addr learned my addr 172.21.15.88:0/4278712991 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:11:56.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.728+0000 7f23bffff700 1 --2- 172.21.15.88:0/4278712991 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f23c01523d0 0x7f23c014c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:56.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.728+0000 7f23c4fcd700 1 --2- 172.21.15.88:0/4278712991 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f23c01533f0 0x7f23c01622b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:56.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.728+0000 7f23bf7fe700 1 -- 172.21.15.88:0/4278712991 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f23c01533f0 msgr2=0x7f23c01622b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:56.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.728+0000 7f23bf7fe700 1 --2- 172.21.15.88:0/4278712991 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f23c01533f0 0x7f23c01622b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:56.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.728+0000 7f23bf7fe700 1 -- 172.21.15.88:0/4278712991 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f23c01523d0 msgr2=0x7f23c014c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:56.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.728+0000 7f23bf7fe700 1 --2- 172.21.15.88:0/4278712991 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f23c01523d0 0x7f23c014c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:56.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.728+0000 7f23bf7fe700 1 -- 172.21.15.88:0/4278712991 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23b8009ae0 con 0x7f23c0153010 2023-10-18T16:11:56.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.728+0000 7f23c4fcd700 1 --2- 172.21.15.88:0/4278712991 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f23c01533f0 0x7f23c01622b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:11:56.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.728+0000 7f23bf7fe700 1 --2- 172.21.15.88:0/4278712991 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f23c0153010 0x7f23c0161dc0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f23b400d890 tx=0x7f23b4008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:11:56.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.728+0000 7f23a7fff700 1 -- 172.21.15.88:0/4278712991 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b4011590 con 0x7f23c0153010 2023-10-18T16:11:56.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.728+0000 7f23a7fff700 1 -- 172.21.15.88:0/4278712991 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23b40116f0 con 0x7f23c0153010 2023-10-18T16:11:56.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.728+0000 7f23a7fff700 1 -- 172.21.15.88:0/4278712991 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b4011a70 con 0x7f23c0153010 2023-10-18T16:11:56.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.728+0000 7f23c6a30700 1 -- 172.21.15.88:0/4278712991 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23c01627a0 con 0x7f23c0153010 2023-10-18T16:11:56.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.728+0000 7f23c6a30700 1 -- 172.21.15.88:0/4278712991 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23c0161320 con 0x7f23c0153010 2023-10-18T16:11:56.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.729+0000 7f23c6a30700 1 -- 172.21.15.88:0/4278712991 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23c0161180 con 0x7f23c0153010 2023-10-18T16:11:56.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.730+0000 7f23a7fff700 1 -- 172.21.15.88:0/4278712991 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f23b4012420 con 0x7f23c0153010 2023-10-18T16:11:56.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.730+0000 7f23a7fff700 1 --2- 172.21.15.88:0/4278712991 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f23ac05f150 0x7f23ac061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:11:56.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.730+0000 7f23a7fff700 1 -- 172.21.15.88:0/4278712991 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6725+0+0 (secure 0 0 0) 0x7f23b4079d60 con 0x7f23c0153010 2023-10-18T16:11:56.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.730+0000 7f23bffff700 1 --2- 172.21.15.88:0/4278712991 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f23ac05f150 0x7f23ac061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:11:56.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.731+0000 7f23bffff700 1 --2- 172.21.15.88:0/4278712991 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f23ac05f150 0x7f23ac061610 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f23b0003920 tx=0x7f23b0000bc0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:11:56.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.734+0000 7f23a7fff700 1 -- 172.21.15.88:0/4278712991 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f23c0161180 con 0x7f23c0153010 2023-10-18T16:11:56.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.941+0000 7f23c6a30700 1 -- 172.21.15.88:0/4278712991 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f23c0161aa0 con 0x7f23c0153010 2023-10-18T16:11:56.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.941+0000 7f23a7fff700 1 -- 172.21.15.88:0/4278712991 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7f23b404ad70 con 0x7f23c0153010 2023-10-18T16:11:56.945 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:11:56.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.943+0000 7f23c6a30700 1 -- 172.21.15.88:0/4278712991 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f23ac05f150 msgr2=0x7f23ac061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:56.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.943+0000 7f23c6a30700 1 --2- 172.21.15.88:0/4278712991 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f23ac05f150 0x7f23ac061610 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f23b0003920 tx=0x7f23b0000bc0).stop 2023-10-18T16:11:56.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.943+0000 7f23c6a30700 1 -- 172.21.15.88:0/4278712991 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f23c0153010 msgr2=0x7f23c0161dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:11:56.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.943+0000 7f23c6a30700 1 --2- 172.21.15.88:0/4278712991 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f23c0153010 0x7f23c0161dc0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f23b400d890 tx=0x7f23b4008040).stop 2023-10-18T16:11:56.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.943+0000 7f23c6a30700 1 -- 172.21.15.88:0/4278712991 shutdown_connections 2023-10-18T16:11:56.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.943+0000 7f23c6a30700 1 --2- 172.21.15.88:0/4278712991 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f23ac05f150 0x7f23ac061610 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:56.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.943+0000 7f23c6a30700 1 --2- 172.21.15.88:0/4278712991 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f23c01533f0 0x7f23c01622b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:56.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.943+0000 7f23c6a30700 1 --2- 172.21.15.88:0/4278712991 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f23c01523d0 0x7f23c014c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:56.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.943+0000 7f23c6a30700 1 --2- 172.21.15.88:0/4278712991 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f23c0153010 0x7f23c0161dc0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:11:56.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.943+0000 7f23c6a30700 1 -- 172.21.15.88:0/4278712991 >> 172.21.15.88:0/4278712991 conn(0x7f23c0069d80 msgr2=0x7f23c00a9a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:11:56.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.943+0000 7f23c6a30700 1 -- 172.21.15.88:0/4278712991 shutdown_connections 2023-10-18T16:11:56.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:11:56.943+0000 7f23c6a30700 1 -- 172.21.15.88:0/4278712991 wait complete. 2023-10-18T16:11:56.958 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:11:59.959 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:12:00.084 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.080+0000 7fdfd981d700 1 Processor -- start 2023-10-18T16:12:00.084 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.080+0000 7fdfd981d700 1 -- start start 2023-10-18T16:12:00.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.081+0000 7fdfd981d700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fdfd414d8c0 0x7fdfd4064440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:00.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.081+0000 7fdfd981d700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdfd4065360 0x7fdfd4064930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:00.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.081+0000 7fdfd981d700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfd4064150 con 0x7fdfd40a9050 2023-10-18T16:12:00.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.081+0000 7fdfd981d700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfd4153a40 con 0x7fdfd414d8c0 2023-10-18T16:12:00.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.081+0000 7fdfd981d700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd4063e70 con 0x7fdfd4065360 2023-10-18T16:12:00.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.081+0000 7fdfd2ffd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fdfd414d8c0 0x7fdfd4064440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:00.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.081+0000 7fdfd2ffd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fdfd414d8c0 0x7fdfd4064440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37884/0 (socket says 172.21.15.88:37884) 2023-10-18T16:12:00.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.081+0000 7fdfd2ffd700 1 -- 172.21.15.88:0/578350402 learned_addr learned my addr 172.21.15.88:0/578350402 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:12:00.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.081+0000 7fdfd37fe700 1 --2- 172.21.15.88:0/578350402 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdfd4065360 0x7fdfd4064930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:00.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.081+0000 7fdfd2ffd700 1 -- 172.21.15.88:0/578350402 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdfd4065360 msgr2=0x7fdfd4064930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:12:00.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.081+0000 7fdfd2ffd700 1 --2- 172.21.15.88:0/578350402 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdfd4065360 0x7fdfd4064930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:00.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.081+0000 7fdfd2ffd700 1 -- 172.21.15.88:0/578350402 >> v1:172.21.15.151:6789/0 conn(0x7fdfd40a9050 legacy=0x7fdfd4153ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:12:00.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.081+0000 7fdfd2ffd700 1 -- 172.21.15.88:0/578350402 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfd40c72c0 con 0x7fdfd414d8c0 2023-10-18T16:12:00.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.081+0000 7fdfd2ffd700 1 --2- 172.21.15.88:0/578350402 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdfd414d8c0 0x7fdfd4064440 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7fdfc4006fd0 tx=0x7fdfc40090f0).ready entity=mon.0 client_cookie=eada101a7949fa32 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:12:00.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.082+0000 7fdfd27fc700 1 -- 172.21.15.88:0/578350402 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfc400b600 con 0x7fdfd414d8c0 2023-10-18T16:12:00.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.082+0000 7fdfd27fc700 1 -- 172.21.15.88:0/578350402 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfc400b760 con 0x7fdfd414d8c0 2023-10-18T16:12:00.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.082+0000 7fdfd27fc700 1 -- 172.21.15.88:0/578350402 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfc400bb00 con 0x7fdfd414d8c0 2023-10-18T16:12:00.090 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.082+0000 7fdfd981d700 1 -- 172.21.15.88:0/578350402 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdfd414d8c0 msgr2=0x7fdfd4064440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:00.090 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.082+0000 7fdfd981d700 1 --2- 172.21.15.88:0/578350402 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdfd414d8c0 0x7fdfd4064440 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7fdfc4006fd0 tx=0x7fdfc40090f0).stop 2023-10-18T16:12:00.090 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.082+0000 7fdfd981d700 1 -- 172.21.15.88:0/578350402 shutdown_connections 2023-10-18T16:12:00.090 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.082+0000 7fdfd981d700 1 --2- 172.21.15.88:0/578350402 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdfd4065360 0x7fdfd4064930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:00.090 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.082+0000 7fdfd981d700 1 --2- 172.21.15.88:0/578350402 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdfd414d8c0 0x7fdfd4064440 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:00.091 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.082+0000 7fdfd981d700 1 -- 172.21.15.88:0/578350402 >> 172.21.15.88:0/578350402 conn(0x7fdfd406a810 msgr2=0x7fdfd40c6a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:12:00.091 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.082+0000 7fdfd981d700 1 -- 172.21.15.88:0/578350402 shutdown_connections 2023-10-18T16:12:00.091 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.082+0000 7fdfd981d700 1 -- 172.21.15.88:0/578350402 wait complete. 2023-10-18T16:12:00.094 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.083+0000 7fdfd981d700 1 Processor -- start 2023-10-18T16:12:00.094 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.084+0000 7fdfd981d700 1 -- start start 2023-10-18T16:12:00.094 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.084+0000 7fdfd981d700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdfd4065360 0x7fdfd4150ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:00.094 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.084+0000 7fdfd981d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdfd40a9050 0x7fdfd40ac070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:00.095 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.084+0000 7fdfd981d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdfd414d8c0 0x7fdfd40bb660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:00.095 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.084+0000 7fdfd981d700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd40ab3b0 con 0x7fdfd414d8c0 2023-10-18T16:12:00.095 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.084+0000 7fdfd981d700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd40abbe0 con 0x7fdfd40a9050 2023-10-18T16:12:00.095 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.084+0000 7fdfd981d700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd40c6730 con 0x7fdfd4065360 2023-10-18T16:12:00.095 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.084+0000 7fdfd37fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdfd414d8c0 0x7fdfd40bb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:00.096 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.084+0000 7fdfd37fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdfd414d8c0 0x7fdfd40bb660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37898/0 (socket says 172.21.15.88:37898) 2023-10-18T16:12:00.096 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.084+0000 7fdfd37fe700 1 -- 172.21.15.88:0/2719492721 learned_addr learned my addr 172.21.15.88:0/2719492721 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:12:00.096 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.084+0000 7fdfca7fc700 1 --2- 172.21.15.88:0/2719492721 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdfd40a9050 0x7fdfd40ac070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:00.096 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.084+0000 7fdfd2ffd700 1 --2- 172.21.15.88:0/2719492721 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdfd4065360 0x7fdfd4150ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:00.096 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.085+0000 7fdfd37fe700 1 -- 172.21.15.88:0/2719492721 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdfd4065360 msgr2=0x7fdfd4150ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:00.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.085+0000 7fdfd37fe700 1 --2- 172.21.15.88:0/2719492721 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdfd4065360 0x7fdfd4150ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:00.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.085+0000 7fdfd37fe700 1 -- 172.21.15.88:0/2719492721 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdfd40a9050 msgr2=0x7fdfd40ac070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:00.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.085+0000 7fdfd37fe700 1 --2- 172.21.15.88:0/2719492721 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdfd40a9050 0x7fdfd40ac070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:00.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.085+0000 7fdfd37fe700 1 -- 172.21.15.88:0/2719492721 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfc4009370 con 0x7fdfd414d8c0 2023-10-18T16:12:00.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.085+0000 7fdfd37fe700 1 --2- 172.21.15.88:0/2719492721 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdfd414d8c0 0x7fdfd40bb660 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7fdfcc009fa0 tx=0x7fdfcc009900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:12:00.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.085+0000 7fdfcb7fe700 1 -- 172.21.15.88:0/2719492721 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfcc00ebc0 con 0x7fdfd414d8c0 2023-10-18T16:12:00.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.085+0000 7fdfcb7fe700 1 -- 172.21.15.88:0/2719492721 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfcc00ed20 con 0x7fdfd414d8c0 2023-10-18T16:12:00.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.085+0000 7fdfd981d700 1 -- 172.21.15.88:0/2719492721 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfd40bbb50 con 0x7fdfd414d8c0 2023-10-18T16:12:00.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.085+0000 7fdfcb7fe700 1 -- 172.21.15.88:0/2719492721 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfcc014610 con 0x7fdfd414d8c0 2023-10-18T16:12:00.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.085+0000 7fdfd981d700 1 -- 172.21.15.88:0/2719492721 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfd40ab590 con 0x7fdfd414d8c0 2023-10-18T16:12:00.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.086+0000 7fdfd981d700 1 -- 172.21.15.88:0/2719492721 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfd40abd90 con 0x7fdfd414d8c0 2023-10-18T16:12:00.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.087+0000 7fdfcb7fe700 1 -- 172.21.15.88:0/2719492721 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7fdfcc01e440 con 0x7fdfd414d8c0 2023-10-18T16:12:00.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.087+0000 7fdfcb7fe700 1 --2- 172.21.15.88:0/2719492721 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fdfb405f120 0x7fdfb40615e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:00.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.087+0000 7fdfcb7fe700 1 -- 172.21.15.88:0/2719492721 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6725+0+0 (secure 0 0 0) 0x7fdfcc07add0 con 0x7fdfd414d8c0 2023-10-18T16:12:00.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.088+0000 7fdfd2ffd700 1 --2- 172.21.15.88:0/2719492721 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fdfb405f120 0x7fdfb40615e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:00.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.088+0000 7fdfd2ffd700 1 --2- 172.21.15.88:0/2719492721 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fdfb405f120 0x7fdfb40615e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fdfc400b3c0 tx=0x7fdfc4017000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:12:00.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.091+0000 7fdfcb7fe700 1 -- 172.21.15.88:0/2719492721 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7fdfcc04be80 con 0x7fdfd414d8c0 2023-10-18T16:12:00.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.292+0000 7fdfd981d700 1 -- 172.21.15.88:0/2719492721 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fdfd406c4e0 con 0x7fdfd414d8c0 2023-10-18T16:12:00.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.292+0000 7fdfcb7fe700 1 -- 172.21.15.88:0/2719492721 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7fdfcc04f4a0 con 0x7fdfd414d8c0 2023-10-18T16:12:00.296 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:12:00.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.294+0000 7fdfd981d700 1 -- 172.21.15.88:0/2719492721 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fdfb405f120 msgr2=0x7fdfb40615e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:00.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.294+0000 7fdfd981d700 1 --2- 172.21.15.88:0/2719492721 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fdfb405f120 0x7fdfb40615e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fdfc400b3c0 tx=0x7fdfc4017000).stop 2023-10-18T16:12:00.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.294+0000 7fdfd981d700 1 -- 172.21.15.88:0/2719492721 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdfd414d8c0 msgr2=0x7fdfd40bb660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:00.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.294+0000 7fdfd981d700 1 --2- 172.21.15.88:0/2719492721 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdfd414d8c0 0x7fdfd40bb660 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7fdfcc009fa0 tx=0x7fdfcc009900).stop 2023-10-18T16:12:00.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.294+0000 7fdfd981d700 1 -- 172.21.15.88:0/2719492721 shutdown_connections 2023-10-18T16:12:00.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.294+0000 7fdfd981d700 1 --2- 172.21.15.88:0/2719492721 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fdfb405f120 0x7fdfb40615e0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:00.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.294+0000 7fdfd981d700 1 --2- 172.21.15.88:0/2719492721 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdfd414d8c0 0x7fdfd40bb660 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:00.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.294+0000 7fdfd981d700 1 --2- 172.21.15.88:0/2719492721 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdfd4065360 0x7fdfd4150ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:00.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.294+0000 7fdfd981d700 1 --2- 172.21.15.88:0/2719492721 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdfd40a9050 0x7fdfd40ac070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:00.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.294+0000 7fdfd981d700 1 -- 172.21.15.88:0/2719492721 >> 172.21.15.88:0/2719492721 conn(0x7fdfd406a810 msgr2=0x7fdfd4153ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:12:00.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.294+0000 7fdfd981d700 1 -- 172.21.15.88:0/2719492721 shutdown_connections 2023-10-18T16:12:00.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:00.294+0000 7fdfd981d700 1 -- 172.21.15.88:0/2719492721 wait complete. 2023-10-18T16:12:00.309 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:12:01.043 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~145s 2023-10-18T16:12:01.043 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~158s 2023-10-18T16:12:01.043 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~191s 2023-10-18T16:12:01.044 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~171s 2023-10-18T16:12:01.044 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~252s 2023-10-18T16:12:01.044 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.f is failed for ~204s 2023-10-18T16:12:03.310 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:12:03.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.432+0000 7fb323568700 1 Processor -- start 2023-10-18T16:12:03.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.432+0000 7fb323568700 1 -- start start 2023-10-18T16:12:03.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.433+0000 7fb323568700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb31c14d8c0 0x7fb31c064440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:03.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.433+0000 7fb323568700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb31c064150 con 0x7fb31c065360 2023-10-18T16:12:03.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.433+0000 7fb323568700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb31c153a40 con 0x7fb31c0a9050 2023-10-18T16:12:03.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.433+0000 7fb323568700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb31c063e70 con 0x7fb31c14d8c0 2023-10-18T16:12:03.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.433+0000 7fb321b05700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fb31c065360 0x7fb31c15acb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:46056/0 (socket says 172.21.15.88:46056) 2023-10-18T16:12:03.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.433+0000 7fb321b05700 1 -- 172.21.15.88:0/3414305341 learned_addr learned my addr 172.21.15.88:0/3414305341 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:12:03.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.433+0000 7fb321304700 1 --2- 172.21.15.88:0/3414305341 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb31c14d8c0 0x7fb31c064440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:03.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.433+0000 7fb313fff700 1 -- 172.21.15.88:0/3414305341 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2459259836 0 0) 0x7fb31c064150 con 0x7fb31c065360 2023-10-18T16:12:03.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.433+0000 7fb313fff700 1 -- 172.21.15.88:0/3414305341 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3080033f0 con 0x7fb31c065360 2023-10-18T16:12:03.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.433+0000 7fb313fff700 1 -- 172.21.15.88:0/3414305341 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3095206333 0 0) 0x7fb31c153a40 con 0x7fb31c0a9050 2023-10-18T16:12:03.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.433+0000 7fb313fff700 1 -- 172.21.15.88:0/3414305341 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb31c064150 con 0x7fb31c0a9050 2023-10-18T16:12:03.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.433+0000 7fb313fff700 1 -- 172.21.15.88:0/3414305341 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 552348435 0 0) 0x7fb3080033f0 con 0x7fb31c065360 2023-10-18T16:12:03.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.433+0000 7fb313fff700 1 -- 172.21.15.88:0/3414305341 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb31c14d8c0 msgr2=0x7fb31c064440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:12:03.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.433+0000 7fb313fff700 1 --2- 172.21.15.88:0/3414305341 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb31c14d8c0 0x7fb31c064440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:03.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.433+0000 7fb313fff700 1 -- 172.21.15.88:0/3414305341 >> v1:172.21.15.151:6789/0 conn(0x7fb31c0a9050 legacy=0x7fb31c153ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:03.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.433+0000 7fb313fff700 1 -- 172.21.15.88:0/3414305341 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb31c0c72c0 con 0x7fb31c065360 2023-10-18T16:12:03.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.433+0000 7fb313fff700 1 -- 172.21.15.88:0/3414305341 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441715075 0 0) 0x7fb3140017f0 con 0x7fb31c065360 2023-10-18T16:12:03.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.433+0000 7fb313fff700 1 -- 172.21.15.88:0/3414305341 >> v1:172.21.15.88:6789/0 conn(0x7fb31c065360 legacy=0x7fb31c15acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:03.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.433+0000 7fb313fff700 1 --2- 172.21.15.88:0/3414305341 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb308003aa0 0x7fb308003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:03.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.433+0000 7fb313fff700 1 --2- 172.21.15.88:0/3414305341 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb308004690 0x7fb308004b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:03.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.434+0000 7fb313fff700 1 --2- 172.21.15.88:0/3414305341 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb31c14d8c0 0x7fb308005060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:03.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.434+0000 7fb313fff700 1 -- 172.21.15.88:0/3414305341 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb31c063e70 con 0x7fb308003aa0 2023-10-18T16:12:03.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.434+0000 7fb313fff700 1 -- 172.21.15.88:0/3414305341 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb30800b990 con 0x7fb308004690 2023-10-18T16:12:03.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.434+0000 7fb313fff700 1 -- 172.21.15.88:0/3414305341 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb30800bb50 con 0x7fb31c14d8c0 2023-10-18T16:12:03.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.434+0000 7fb321304700 1 --2- 172.21.15.88:0/3414305341 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb308003aa0 0x7fb308003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:03.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.434+0000 7fb320b03700 1 --2- 172.21.15.88:0/3414305341 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb308004690 0x7fb308004b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:03.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.434+0000 7fb321b05700 1 --2- 172.21.15.88:0/3414305341 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb31c14d8c0 0x7fb308005060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:03.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.434+0000 7fb321304700 1 -- 172.21.15.88:0/3414305341 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb31c14d8c0 msgr2=0x7fb308005060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:12:03.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.434+0000 7fb321304700 1 --2- 172.21.15.88:0/3414305341 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb31c14d8c0 0x7fb308005060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:03.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.434+0000 7fb321304700 1 -- 172.21.15.88:0/3414305341 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb308004690 msgr2=0x7fb308004b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:12:03.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.434+0000 7fb321304700 1 --2- 172.21.15.88:0/3414305341 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb308004690 0x7fb308004b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:03.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.434+0000 7fb321304700 1 -- 172.21.15.88:0/3414305341 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3080033f0 con 0x7fb308003aa0 2023-10-18T16:12:03.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.434+0000 7fb321304700 1 --2- 172.21.15.88:0/3414305341 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb308003aa0 0x7fb308003ea0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7fb318009a80 tx=0x7fb318009060).ready entity=mon.0 client_cookie=d7da1451fd73439a server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:12:03.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.434+0000 7fb313fff700 1 -- 172.21.15.88:0/3414305341 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb318009430 con 0x7fb308003aa0 2023-10-18T16:12:03.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.434+0000 7fb313fff700 1 -- 172.21.15.88:0/3414305341 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb31800e640 con 0x7fb308003aa0 2023-10-18T16:12:03.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.434+0000 7fb313fff700 1 -- 172.21.15.88:0/3414305341 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb31800e9c0 con 0x7fb308003aa0 2023-10-18T16:12:03.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.435+0000 7fb323568700 1 -- 172.21.15.88:0/3414305341 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb308003aa0 msgr2=0x7fb308003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:03.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.435+0000 7fb323568700 1 --2- 172.21.15.88:0/3414305341 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb308003aa0 0x7fb308003ea0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7fb318009a80 tx=0x7fb318009060).stop 2023-10-18T16:12:03.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.435+0000 7fb321b05700 1 -- 172.21.15.88:0/3414305341 reap_dead start 2023-10-18T16:12:03.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.435+0000 7fb323568700 1 -- 172.21.15.88:0/3414305341 shutdown_connections 2023-10-18T16:12:03.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.435+0000 7fb323568700 1 -- 172.21.15.88:0/3414305341 >> 172.21.15.88:0/3414305341 conn(0x7fb31c06a810 msgr2=0x7fb31c0c6a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:12:03.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.435+0000 7fb323568700 1 -- 172.21.15.88:0/3414305341 shutdown_connections 2023-10-18T16:12:03.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.435+0000 7fb323568700 1 -- 172.21.15.88:0/3414305341 wait complete. 2023-10-18T16:12:03.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.436+0000 7fb323568700 1 Processor -- start 2023-10-18T16:12:03.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.436+0000 7fb323568700 1 -- start start 2023-10-18T16:12:03.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.437+0000 7fb323568700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb308003aa0 0x7fb31c150e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:03.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.437+0000 7fb323568700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb31c150010 0x7fb31c0bbdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:03.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.437+0000 7fb323568700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb31c0bc2a0 0x7fb31c148bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:03.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.437+0000 7fb323568700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb31c0ac4d0 con 0x7fb31c150010 2023-10-18T16:12:03.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.437+0000 7fb323568700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb31c0aca00 con 0x7fb308003aa0 2023-10-18T16:12:03.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.437+0000 7fb323568700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb31c160d80 con 0x7fb31c0bc2a0 2023-10-18T16:12:03.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.437+0000 7fb320b03700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb31c150010 0x7fb31c0bbdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:03.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.437+0000 7fb320b03700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb31c150010 0x7fb31c0bbdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37914/0 (socket says 172.21.15.88:37914) 2023-10-18T16:12:03.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.437+0000 7fb320b03700 1 -- 172.21.15.88:0/2081535578 learned_addr learned my addr 172.21.15.88:0/2081535578 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:12:03.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.437+0000 7fb321304700 1 --2- 172.21.15.88:0/2081535578 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb308003aa0 0x7fb31c150e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:03.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.437+0000 7fb321b05700 1 --2- 172.21.15.88:0/2081535578 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb31c0bc2a0 0x7fb31c148bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:03.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.437+0000 7fb320b03700 1 -- 172.21.15.88:0/2081535578 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb31c0bc2a0 msgr2=0x7fb31c148bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:03.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.437+0000 7fb320b03700 1 --2- 172.21.15.88:0/2081535578 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb31c0bc2a0 0x7fb31c148bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:03.454 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.437+0000 7fb320b03700 1 -- 172.21.15.88:0/2081535578 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb308003aa0 msgr2=0x7fb31c150e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:03.454 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.437+0000 7fb320b03700 1 --2- 172.21.15.88:0/2081535578 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb308003aa0 0x7fb31c150e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:03.454 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.437+0000 7fb320b03700 1 -- 172.21.15.88:0/2081535578 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb31c161210 con 0x7fb31c150010 2023-10-18T16:12:03.454 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.437+0000 7fb321b05700 1 --2- 172.21.15.88:0/2081535578 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb31c0bc2a0 0x7fb31c148bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:12:03.454 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.437+0000 7fb320b03700 1 --2- 172.21.15.88:0/2081535578 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb31c150010 0x7fb31c0bbdb0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7fb30c002a80 tx=0x7fb30c007bb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:12:03.454 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.437+0000 7fb3117fa700 1 -- 172.21.15.88:0/2081535578 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb30c007240 con 0x7fb31c150010 2023-10-18T16:12:03.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.437+0000 7fb3117fa700 1 -- 172.21.15.88:0/2081535578 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb30c0073a0 con 0x7fb31c150010 2023-10-18T16:12:03.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.437+0000 7fb323568700 1 -- 172.21.15.88:0/2081535578 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb31c0ac720 con 0x7fb31c150010 2023-10-18T16:12:03.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.437+0000 7fb3117fa700 1 -- 172.21.15.88:0/2081535578 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb30c0066a0 con 0x7fb31c150010 2023-10-18T16:12:03.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.438+0000 7fb323568700 1 -- 172.21.15.88:0/2081535578 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb31c160730 con 0x7fb31c150010 2023-10-18T16:12:03.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.438+0000 7fb323568700 1 -- 172.21.15.88:0/2081535578 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb31c160f30 con 0x7fb31c150010 2023-10-18T16:12:03.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.439+0000 7fb3117fa700 1 -- 172.21.15.88:0/2081535578 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7fb30c006a50 con 0x7fb31c150010 2023-10-18T16:12:03.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.439+0000 7fb3117fa700 1 --2- 172.21.15.88:0/2081535578 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fb30005f180 0x7fb300061640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:03.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.439+0000 7fb3117fa700 1 -- 172.21.15.88:0/2081535578 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6725+0+0 (secure 0 0 0) 0x7fb30c079ee0 con 0x7fb31c150010 2023-10-18T16:12:03.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.439+0000 7fb321304700 1 --2- 172.21.15.88:0/2081535578 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fb30005f180 0x7fb300061640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:03.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.440+0000 7fb321304700 1 --2- 172.21.15.88:0/2081535578 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fb30005f180 0x7fb300061640 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7fb318006d50 tx=0x7fb31801a000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:12:03.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.443+0000 7fb3117fa700 1 -- 172.21.15.88:0/2081535578 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7fb30c049ac0 con 0x7fb31c150010 2023-10-18T16:12:03.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.650+0000 7fb323568700 1 -- 172.21.15.88:0/2081535578 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb31c06c4e0 con 0x7fb31c150010 2023-10-18T16:12:03.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.650+0000 7fb3117fa700 1 -- 172.21.15.88:0/2081535578 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7fb31c160f30 con 0x7fb31c150010 2023-10-18T16:12:03.654 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:12:03.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.652+0000 7fb323568700 1 -- 172.21.15.88:0/2081535578 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fb30005f180 msgr2=0x7fb300061640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:03.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.652+0000 7fb323568700 1 --2- 172.21.15.88:0/2081535578 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fb30005f180 0x7fb300061640 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7fb318006d50 tx=0x7fb31801a000).stop 2023-10-18T16:12:03.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.652+0000 7fb323568700 1 -- 172.21.15.88:0/2081535578 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb31c150010 msgr2=0x7fb31c0bbdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:03.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.652+0000 7fb323568700 1 --2- 172.21.15.88:0/2081535578 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb31c150010 0x7fb31c0bbdb0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7fb30c002a80 tx=0x7fb30c007bb0).stop 2023-10-18T16:12:03.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.652+0000 7fb323568700 1 -- 172.21.15.88:0/2081535578 shutdown_connections 2023-10-18T16:12:03.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.652+0000 7fb323568700 1 --2- 172.21.15.88:0/2081535578 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fb30005f180 0x7fb300061640 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:03.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.652+0000 7fb323568700 1 --2- 172.21.15.88:0/2081535578 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb31c0bc2a0 0x7fb31c148bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:03.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.652+0000 7fb323568700 1 --2- 172.21.15.88:0/2081535578 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb308003aa0 0x7fb31c150e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:03.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.652+0000 7fb323568700 1 --2- 172.21.15.88:0/2081535578 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb31c150010 0x7fb31c0bbdb0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:03.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.652+0000 7fb323568700 1 -- 172.21.15.88:0/2081535578 >> 172.21.15.88:0/2081535578 conn(0x7fb31c06a810 msgr2=0x7fb31c0c14b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:12:03.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.652+0000 7fb323568700 1 -- 172.21.15.88:0/2081535578 shutdown_connections 2023-10-18T16:12:03.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:03.652+0000 7fb323568700 1 -- 172.21.15.88:0/2081535578 wait complete. 2023-10-18T16:12:03.667 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:12:06.668 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:12:06.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.787+0000 7ffaa9155700 1 Processor -- start 2023-10-18T16:12:06.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.787+0000 7ffaa9155700 1 -- start start 2023-10-18T16:12:06.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.787+0000 7ffaa9155700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ffaa414d8c0 0x7ffaa4064440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:06.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.787+0000 7ffaa9155700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ffaa40a9050 0x7ffaa4064930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:06.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.787+0000 7ffaa9155700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaa4064150 con 0x7ffaa4065360 2023-10-18T16:12:06.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.787+0000 7ffaa9155700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffaa4153a40 con 0x7ffaa414d8c0 2023-10-18T16:12:06.794 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.787+0000 7ffaa9155700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa4063e70 con 0x7ffaa40a9050 2023-10-18T16:12:06.794 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.788+0000 7ffaa2d9d700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ffaa414d8c0 0x7ffaa4064440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:06.794 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.788+0000 7ffaa359e700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7ffaa4065360 0x7ffaa4153ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:46064/0 (socket says 172.21.15.88:46064) 2023-10-18T16:12:06.794 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.788+0000 7ffaa359e700 1 -- 172.21.15.88:0/2326228482 learned_addr learned my addr 172.21.15.88:0/2326228482 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:12:06.794 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.788+0000 7ffa9a59c700 1 --2- 172.21.15.88:0/2326228482 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ffaa40a9050 0x7ffaa4064930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:06.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.788+0000 7ffaa2d9d700 1 -- 172.21.15.88:0/2326228482 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ffaa40a9050 msgr2=0x7ffaa4064930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:12:06.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.788+0000 7ffaa2d9d700 1 --2- 172.21.15.88:0/2326228482 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ffaa40a9050 0x7ffaa4064930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:06.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.788+0000 7ffaa2d9d700 1 -- 172.21.15.88:0/2326228482 >> v1:172.21.15.88:6789/0 conn(0x7ffaa4065360 legacy=0x7ffaa4153ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:06.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.788+0000 7ffaa259c700 1 -- 172.21.15.88:0/2326228482 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1222173190 0 0) 0x7ffaa4064150 con 0x7ffaa4065360 2023-10-18T16:12:06.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.788+0000 7ffaa2d9d700 1 -- 172.21.15.88:0/2326228482 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaa40c72c0 con 0x7ffaa414d8c0 2023-10-18T16:12:06.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.788+0000 7ffa9a59c700 1 --2- 172.21.15.88:0/2326228482 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ffaa40a9050 0x7ffaa4064930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-10-18T16:12:06.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.788+0000 7ffaa2d9d700 1 --2- 172.21.15.88:0/2326228482 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffaa414d8c0 0x7ffaa4064440 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7ffa94009760 tx=0x7ffa94009060).ready entity=mon.0 client_cookie=c1144f97b49cded5 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:12:06.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.788+0000 7ffaa259c700 1 -- 172.21.15.88:0/2326228482 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa940090d0 con 0x7ffaa414d8c0 2023-10-18T16:12:06.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.788+0000 7ffaa259c700 1 -- 172.21.15.88:0/2326228482 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa9400b580 con 0x7ffaa414d8c0 2023-10-18T16:12:06.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.788+0000 7ffaa259c700 1 -- 172.21.15.88:0/2326228482 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa9400b900 con 0x7ffaa414d8c0 2023-10-18T16:12:06.797 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.789+0000 7ffaa9155700 1 -- 172.21.15.88:0/2326228482 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffaa414d8c0 msgr2=0x7ffaa4064440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:06.797 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.789+0000 7ffaa9155700 1 --2- 172.21.15.88:0/2326228482 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffaa414d8c0 0x7ffaa4064440 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7ffa94009760 tx=0x7ffa94009060).stop 2023-10-18T16:12:06.797 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.789+0000 7ffaa9155700 1 -- 172.21.15.88:0/2326228482 shutdown_connections 2023-10-18T16:12:06.797 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.789+0000 7ffaa9155700 1 --2- 172.21.15.88:0/2326228482 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffaa414d8c0 0x7ffaa4064440 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:06.797 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.789+0000 7ffaa9155700 1 --2- 172.21.15.88:0/2326228482 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ffaa40a9050 0x7ffaa4064930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:06.797 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.789+0000 7ffaa9155700 1 -- 172.21.15.88:0/2326228482 >> 172.21.15.88:0/2326228482 conn(0x7ffaa406a810 msgr2=0x7ffaa40c6a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:12:06.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.789+0000 7ffaa9155700 1 -- 172.21.15.88:0/2326228482 shutdown_connections 2023-10-18T16:12:06.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.789+0000 7ffaa9155700 1 -- 172.21.15.88:0/2326228482 wait complete. 2023-10-18T16:12:06.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.790+0000 7ffaa9155700 1 Processor -- start 2023-10-18T16:12:06.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.790+0000 7ffaa9155700 1 -- start start 2023-10-18T16:12:06.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.791+0000 7ffaa9155700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffaa4065360 0x7ffaa4160340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:06.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.791+0000 7ffaa9155700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ffaa40a9050 0x7ffaa414e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:06.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.791+0000 7ffaa9155700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffaa414d8c0 0x7ffaa40bcba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:06.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.791+0000 7ffaa9155700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa414dca0 con 0x7ffaa414d8c0 2023-10-18T16:12:06.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.791+0000 7ffaa9155700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa414e260 con 0x7ffaa4065360 2023-10-18T16:12:06.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.791+0000 7ffaa9155700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa4160b10 con 0x7ffaa40a9050 2023-10-18T16:12:06.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.791+0000 7ffaa359e700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffaa414d8c0 0x7ffaa40bcba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:06.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.791+0000 7ffaa359e700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffaa414d8c0 0x7ffaa40bcba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37940/0 (socket says 172.21.15.88:37940) 2023-10-18T16:12:06.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.791+0000 7ffaa359e700 1 -- 172.21.15.88:0/743764432 learned_addr learned my addr 172.21.15.88:0/743764432 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:12:06.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.791+0000 7ffaa2d9d700 1 --2- 172.21.15.88:0/743764432 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffaa4065360 0x7ffaa4160340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:06.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.791+0000 7ffa9a59c700 1 --2- 172.21.15.88:0/743764432 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ffaa40a9050 0x7ffaa414e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:06.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.791+0000 7ffaa359e700 1 -- 172.21.15.88:0/743764432 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ffaa40a9050 msgr2=0x7ffaa414e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:06.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.791+0000 7ffaa359e700 1 --2- 172.21.15.88:0/743764432 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ffaa40a9050 0x7ffaa414e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:06.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.791+0000 7ffaa359e700 1 -- 172.21.15.88:0/743764432 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffaa4065360 msgr2=0x7ffaa4160340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:06.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.791+0000 7ffaa359e700 1 --2- 172.21.15.88:0/743764432 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffaa4065360 0x7ffaa4160340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:06.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.791+0000 7ffaa359e700 1 -- 172.21.15.88:0/743764432 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa940092a0 con 0x7ffaa414d8c0 2023-10-18T16:12:06.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.791+0000 7ffaa2d9d700 1 --2- 172.21.15.88:0/743764432 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffaa4065360 0x7ffaa4160340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:12:06.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.791+0000 7ffaa359e700 1 --2- 172.21.15.88:0/743764432 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffaa414d8c0 0x7ffaa40bcba0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7ffa9c00c810 tx=0x7ffa9c007f90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:12:06.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.791+0000 7ffa9b7fe700 1 -- 172.21.15.88:0/743764432 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa9c005d30 con 0x7ffaa414d8c0 2023-10-18T16:12:06.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.791+0000 7ffa9b7fe700 1 -- 172.21.15.88:0/743764432 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa9c005e90 con 0x7ffaa414d8c0 2023-10-18T16:12:06.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.791+0000 7ffaa9155700 1 -- 172.21.15.88:0/743764432 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaa4160df0 con 0x7ffaa414d8c0 2023-10-18T16:12:06.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.791+0000 7ffa9b7fe700 1 -- 172.21.15.88:0/743764432 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa9c007320 con 0x7ffaa414d8c0 2023-10-18T16:12:06.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.791+0000 7ffaa9155700 1 -- 172.21.15.88:0/743764432 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaa414def0 con 0x7ffaa414d8c0 2023-10-18T16:12:06.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.792+0000 7ffaa9155700 1 -- 172.21.15.88:0/743764432 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaa406c4e0 con 0x7ffaa414d8c0 2023-10-18T16:12:06.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.793+0000 7ffa9b7fe700 1 -- 172.21.15.88:0/743764432 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7ffa9c007710 con 0x7ffaa414d8c0 2023-10-18T16:12:06.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.793+0000 7ffa9b7fe700 1 --2- 172.21.15.88:0/743764432 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7ffa8405f150 0x7ffa84061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:06.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.793+0000 7ffa9b7fe700 1 -- 172.21.15.88:0/743764432 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6725+0+0 (secure 0 0 0) 0x7ffa9c07a540 con 0x7ffaa414d8c0 2023-10-18T16:12:06.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.793+0000 7ffaa2d9d700 1 --2- 172.21.15.88:0/743764432 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7ffa8405f150 0x7ffa84061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:06.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.794+0000 7ffaa2d9d700 1 --2- 172.21.15.88:0/743764432 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7ffa8405f150 0x7ffa84061610 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7ffa9400b340 tx=0x7ffa94017000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:12:06.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.797+0000 7ffa9b7fe700 1 -- 172.21.15.88:0/743764432 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7ffa9c04b550 con 0x7ffaa414d8c0 2023-10-18T16:12:06.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.996+0000 7ffaa9155700 1 -- 172.21.15.88:0/743764432 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ffaa414e3a0 con 0x7ffaa414d8c0 2023-10-18T16:12:07.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.996+0000 7ffa9b7fe700 1 -- 172.21.15.88:0/743764432 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7ffa9c04eb70 con 0x7ffaa414d8c0 2023-10-18T16:12:07.000 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:12:07.002 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.998+0000 7ffaa9155700 1 -- 172.21.15.88:0/743764432 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7ffa8405f150 msgr2=0x7ffa84061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:07.002 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.998+0000 7ffaa9155700 1 --2- 172.21.15.88:0/743764432 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7ffa8405f150 0x7ffa84061610 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7ffa9400b340 tx=0x7ffa94017000).stop 2023-10-18T16:12:07.002 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.998+0000 7ffaa9155700 1 -- 172.21.15.88:0/743764432 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffaa414d8c0 msgr2=0x7ffaa40bcba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:07.002 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.998+0000 7ffaa9155700 1 --2- 172.21.15.88:0/743764432 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffaa414d8c0 0x7ffaa40bcba0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7ffa9c00c810 tx=0x7ffa9c007f90).stop 2023-10-18T16:12:07.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.998+0000 7ffaa9155700 1 -- 172.21.15.88:0/743764432 shutdown_connections 2023-10-18T16:12:07.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.998+0000 7ffaa9155700 1 --2- 172.21.15.88:0/743764432 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7ffa8405f150 0x7ffa84061610 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:07.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.998+0000 7ffaa9155700 1 --2- 172.21.15.88:0/743764432 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffaa414d8c0 0x7ffaa40bcba0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:07.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.998+0000 7ffaa9155700 1 --2- 172.21.15.88:0/743764432 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffaa4065360 0x7ffaa4160340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:07.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.998+0000 7ffaa9155700 1 --2- 172.21.15.88:0/743764432 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ffaa40a9050 0x7ffaa414e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:07.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.998+0000 7ffaa9155700 1 -- 172.21.15.88:0/743764432 >> 172.21.15.88:0/743764432 conn(0x7ffaa406a810 msgr2=0x7ffaa41529a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:12:07.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.998+0000 7ffaa9155700 1 -- 172.21.15.88:0/743764432 shutdown_connections 2023-10-18T16:12:07.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:06.998+0000 7ffaa9155700 1 -- 172.21.15.88:0/743764432 wait complete. 2023-10-18T16:12:07.013 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:12:07.354 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~151s 2023-10-18T16:12:07.355 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~164s 2023-10-18T16:12:07.355 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~197s 2023-10-18T16:12:07.355 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~177s 2023-10-18T16:12:07.355 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~258s 2023-10-18T16:12:07.355 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.f is failed for ~211s 2023-10-18T16:12:08.787 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-18T16:12:08.791 DEBUG:teuthology.orchestra.run.smithi151:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-18T16:12:10.015 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:12:10.136 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.132+0000 7fd5e614d700 1 Processor -- start 2023-10-18T16:12:10.136 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.132+0000 7fd5e614d700 1 -- start start 2023-10-18T16:12:10.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.133+0000 7fd5e614d700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd5e014d890 0x7fd5e0150ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:10.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.133+0000 7fd5e614d700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5e0069f00 con 0x7fd5e00c3020 2023-10-18T16:12:10.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.133+0000 7fd5e614d700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5e006a1e0 con 0x7fd5e014dc70 2023-10-18T16:12:10.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.133+0000 7fd5e614d700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5e01513c0 con 0x7fd5e014d890 2023-10-18T16:12:10.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.133+0000 7fd5deffd700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fd5e00c3020 0x7fd5e00a8660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:40362/0 (socket says 172.21.15.88:40362) 2023-10-18T16:12:10.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.133+0000 7fd5deffd700 1 -- 172.21.15.88:0/1728151786 learned_addr learned my addr 172.21.15.88:0/1728151786 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:12:10.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.134+0000 7fd5df7fe700 1 --2- 172.21.15.88:0/1728151786 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd5e014d890 0x7fd5e0150ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:10.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.134+0000 7fd5de7fc700 1 -- 172.21.15.88:0/1728151786 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4034270987 0 0) 0x7fd5e0069f00 con 0x7fd5e00c3020 2023-10-18T16:12:10.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.134+0000 7fd5de7fc700 1 -- 172.21.15.88:0/1728151786 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5c80033f0 con 0x7fd5e00c3020 2023-10-18T16:12:10.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.134+0000 7fd5de7fc700 1 -- 172.21.15.88:0/1728151786 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1635218012 0 0) 0x7fd5c80033f0 con 0x7fd5e00c3020 2023-10-18T16:12:10.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.134+0000 7fd5de7fc700 1 -- 172.21.15.88:0/1728151786 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd5e014d890 msgr2=0x7fd5e0150ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:12:10.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.134+0000 7fd5de7fc700 1 --2- 172.21.15.88:0/1728151786 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd5e014d890 0x7fd5e0150ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:10.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.134+0000 7fd5de7fc700 1 -- 172.21.15.88:0/1728151786 >> v1:172.21.15.151:6789/0 conn(0x7fd5e014dc70 legacy=0x7fd5e0064920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:10.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.134+0000 7fd5de7fc700 1 -- 172.21.15.88:0/1728151786 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5e006a4c0 con 0x7fd5e00c3020 2023-10-18T16:12:10.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.134+0000 7fd5de7fc700 1 -- 172.21.15.88:0/1728151786 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441715075 0 0) 0x7fd5d4001470 con 0x7fd5e00c3020 2023-10-18T16:12:10.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.134+0000 7fd5de7fc700 1 -- 172.21.15.88:0/1728151786 >> v1:172.21.15.88:6789/0 conn(0x7fd5e00c3020 legacy=0x7fd5e00a8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:10.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.134+0000 7fd5de7fc700 1 --2- 172.21.15.88:0/1728151786 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd5e014d890 0x7fd5c8003910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:10.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.134+0000 7fd5de7fc700 1 --2- 172.21.15.88:0/1728151786 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd5c8003e00 0x7fd5c8004690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:10.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.134+0000 7fd5de7fc700 1 --2- 172.21.15.88:0/1728151786 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd5c8004b80 0x7fd5c8005000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:10.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.134+0000 7fd5de7fc700 1 -- 172.21.15.88:0/1728151786 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5e01513c0 con 0x7fd5c8003e00 2023-10-18T16:12:10.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.134+0000 7fd5de7fc700 1 -- 172.21.15.88:0/1728151786 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5c800b8e0 con 0x7fd5c8004b80 2023-10-18T16:12:10.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.134+0000 7fd5de7fc700 1 -- 172.21.15.88:0/1728151786 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5c800baa0 con 0x7fd5e014d890 2023-10-18T16:12:10.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.134+0000 7fd5deffd700 1 --2- 172.21.15.88:0/1728151786 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd5c8003e00 0x7fd5c8004690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:10.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.134+0000 7fd5df7fe700 1 --2- 172.21.15.88:0/1728151786 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd5e014d890 0x7fd5c8003910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:10.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.134+0000 7fd5dffff700 1 --2- 172.21.15.88:0/1728151786 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd5c8004b80 0x7fd5c8005000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:10.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.134+0000 7fd5deffd700 1 -- 172.21.15.88:0/1728151786 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd5e014d890 msgr2=0x7fd5c8003910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:12:10.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.134+0000 7fd5deffd700 1 --2- 172.21.15.88:0/1728151786 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd5e014d890 0x7fd5c8003910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:10.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.134+0000 7fd5deffd700 1 -- 172.21.15.88:0/1728151786 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd5c8004b80 msgr2=0x7fd5c8005000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:12:10.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.134+0000 7fd5deffd700 1 --2- 172.21.15.88:0/1728151786 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd5c8004b80 0x7fd5c8005000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:10.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.134+0000 7fd5deffd700 1 -- 172.21.15.88:0/1728151786 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5c80033f0 con 0x7fd5c8003e00 2023-10-18T16:12:10.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.135+0000 7fd5deffd700 1 --2- 172.21.15.88:0/1728151786 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd5c8003e00 0x7fd5c8004690 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7fd5e00840a0 tx=0x7fd5e00a8580).ready entity=mon.0 client_cookie=9495a52286c4c76d server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:12:10.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.135+0000 7fd5de7fc700 1 -- 172.21.15.88:0/1728151786 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5d4007a60 con 0x7fd5c8003e00 2023-10-18T16:12:10.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.135+0000 7fd5de7fc700 1 -- 172.21.15.88:0/1728151786 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5d400ca50 con 0x7fd5c8003e00 2023-10-18T16:12:10.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.135+0000 7fd5de7fc700 1 -- 172.21.15.88:0/1728151786 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5d400cdd0 con 0x7fd5c8003e00 2023-10-18T16:12:10.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.135+0000 7fd5e614d700 1 -- 172.21.15.88:0/1728151786 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd5c8003e00 msgr2=0x7fd5c8004690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:10.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.135+0000 7fd5e614d700 1 --2- 172.21.15.88:0/1728151786 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd5c8003e00 0x7fd5c8004690 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7fd5e00840a0 tx=0x7fd5e00a8580).stop 2023-10-18T16:12:10.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.135+0000 7fd5dffff700 1 -- 172.21.15.88:0/1728151786 reap_dead start 2023-10-18T16:12:10.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.135+0000 7fd5e614d700 1 -- 172.21.15.88:0/1728151786 shutdown_connections 2023-10-18T16:12:10.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.135+0000 7fd5e614d700 1 -- 172.21.15.88:0/1728151786 >> 172.21.15.88:0/1728151786 conn(0x7fd5e00c6a50 msgr2=0x7fd5e00c6e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:12:10.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.135+0000 7fd5e614d700 1 -- 172.21.15.88:0/1728151786 shutdown_connections 2023-10-18T16:12:10.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.135+0000 7fd5e614d700 1 -- 172.21.15.88:0/1728151786 wait complete. 2023-10-18T16:12:10.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.137+0000 7fd5e614d700 1 Processor -- start 2023-10-18T16:12:10.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.137+0000 7fd5e614d700 1 -- start start 2023-10-18T16:12:10.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.137+0000 7fd5e614d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd5c8003e00 0x7fd5e014f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:10.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.137+0000 7fd5e614d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd5e00bddc0 0x7fd5e00bf6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:10.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.138+0000 7fd5e614d700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd5e0163a20 0x7fd5e0163ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:10.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.138+0000 7fd5e614d700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5e00bfbe0 con 0x7fd5e00bddc0 2023-10-18T16:12:10.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.138+0000 7fd5e614d700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5e00bf0f0 con 0x7fd5c8003e00 2023-10-18T16:12:10.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.138+0000 7fd5e614d700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5e00a9d70 con 0x7fd5e0163a20 2023-10-18T16:12:10.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.138+0000 7fd5deffd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd5e00bddc0 0x7fd5e00bf6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:10.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.138+0000 7fd5deffd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd5e00bddc0 0x7fd5e00bf6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41238/0 (socket says 172.21.15.88:41238) 2023-10-18T16:12:10.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.138+0000 7fd5deffd700 1 -- 172.21.15.88:0/1039333987 learned_addr learned my addr 172.21.15.88:0/1039333987 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:12:10.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.138+0000 7fd5df7fe700 1 --2- 172.21.15.88:0/1039333987 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd5c8003e00 0x7fd5e014f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:10.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.138+0000 7fd5dffff700 1 --2- 172.21.15.88:0/1039333987 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd5e0163a20 0x7fd5e0163ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:10.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.138+0000 7fd5deffd700 1 -- 172.21.15.88:0/1039333987 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd5e0163a20 msgr2=0x7fd5e0163ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:10.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.138+0000 7fd5deffd700 1 --2- 172.21.15.88:0/1039333987 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd5e0163a20 0x7fd5e0163ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:10.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.138+0000 7fd5deffd700 1 -- 172.21.15.88:0/1039333987 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd5c8003e00 msgr2=0x7fd5e014f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:10.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.138+0000 7fd5deffd700 1 --2- 172.21.15.88:0/1039333987 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd5c8003e00 0x7fd5e014f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:10.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.138+0000 7fd5deffd700 1 -- 172.21.15.88:0/1039333987 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5e00a9fc0 con 0x7fd5e00bddc0 2023-10-18T16:12:10.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.138+0000 7fd5dffff700 1 --2- 172.21.15.88:0/1039333987 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd5e0163a20 0x7fd5e0163ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:12:10.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.138+0000 7fd5deffd700 1 --2- 172.21.15.88:0/1039333987 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd5e00bddc0 0x7fd5e00bf6f0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7fd5e00c2eb0 tx=0x7fd5d4015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:12:10.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.138+0000 7fd5c77fe700 1 -- 172.21.15.88:0/1039333987 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5d4001430 con 0x7fd5e00bddc0 2023-10-18T16:12:10.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.138+0000 7fd5c77fe700 1 -- 172.21.15.88:0/1039333987 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5d40074f0 con 0x7fd5e00bddc0 2023-10-18T16:12:10.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.138+0000 7fd5e614d700 1 -- 172.21.15.88:0/1039333987 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5e014eec0 con 0x7fd5e00bddc0 2023-10-18T16:12:10.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.138+0000 7fd5c77fe700 1 -- 172.21.15.88:0/1039333987 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5d40048e0 con 0x7fd5e00bddc0 2023-10-18T16:12:10.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.138+0000 7fd5e614d700 1 -- 172.21.15.88:0/1039333987 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5e014f480 con 0x7fd5e00bddc0 2023-10-18T16:12:10.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.139+0000 7fd5e614d700 1 -- 172.21.15.88:0/1039333987 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5e014ed20 con 0x7fd5e00bddc0 2023-10-18T16:12:10.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.140+0000 7fd5c77fe700 1 -- 172.21.15.88:0/1039333987 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7fd5d4004c50 con 0x7fd5e00bddc0 2023-10-18T16:12:10.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.140+0000 7fd5c77fe700 1 --2- 172.21.15.88:0/1039333987 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fd5cc05f150 0x7fd5cc061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:10.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.140+0000 7fd5c77fe700 1 -- 172.21.15.88:0/1039333987 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6725+0+0 (secure 0 0 0) 0x7fd5d407b100 con 0x7fd5e00bddc0 2023-10-18T16:12:10.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.140+0000 7fd5df7fe700 1 --2- 172.21.15.88:0/1039333987 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fd5cc05f150 0x7fd5cc061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:10.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.141+0000 7fd5df7fe700 1 --2- 172.21.15.88:0/1039333987 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fd5cc05f150 0x7fd5cc061610 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7fd5d00094d0 tx=0x7fd5d000b040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:12:10.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.144+0000 7fd5c77fe700 1 -- 172.21.15.88:0/1039333987 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7fd5e014ed20 con 0x7fd5e00bddc0 2023-10-18T16:12:10.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.344+0000 7fd5e614d700 1 -- 172.21.15.88:0/1039333987 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd5e014ed20 con 0x7fd5e00bddc0 2023-10-18T16:12:10.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.345+0000 7fd5c77fe700 1 -- 172.21.15.88:0/1039333987 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7fd5e014ed20 con 0x7fd5e00bddc0 2023-10-18T16:12:10.349 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:12:10.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.347+0000 7fd5e614d700 1 -- 172.21.15.88:0/1039333987 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fd5cc05f150 msgr2=0x7fd5cc061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:10.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.347+0000 7fd5e614d700 1 --2- 172.21.15.88:0/1039333987 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fd5cc05f150 0x7fd5cc061610 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7fd5d00094d0 tx=0x7fd5d000b040).stop 2023-10-18T16:12:10.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.347+0000 7fd5e614d700 1 -- 172.21.15.88:0/1039333987 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd5e00bddc0 msgr2=0x7fd5e00bf6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:10.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.347+0000 7fd5e614d700 1 --2- 172.21.15.88:0/1039333987 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd5e00bddc0 0x7fd5e00bf6f0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7fd5e00c2eb0 tx=0x7fd5d4015000).stop 2023-10-18T16:12:10.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.347+0000 7fd5e614d700 1 -- 172.21.15.88:0/1039333987 shutdown_connections 2023-10-18T16:12:10.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.347+0000 7fd5e614d700 1 --2- 172.21.15.88:0/1039333987 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fd5cc05f150 0x7fd5cc061610 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:10.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.347+0000 7fd5e614d700 1 --2- 172.21.15.88:0/1039333987 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd5e0163a20 0x7fd5e0163ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:10.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.347+0000 7fd5e614d700 1 --2- 172.21.15.88:0/1039333987 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd5c8003e00 0x7fd5e014f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:10.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.347+0000 7fd5e614d700 1 --2- 172.21.15.88:0/1039333987 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd5e00bddc0 0x7fd5e00bf6f0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:10.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.347+0000 7fd5e614d700 1 -- 172.21.15.88:0/1039333987 >> 172.21.15.88:0/1039333987 conn(0x7fd5e00c6a50 msgr2=0x7fd5e00c55e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:12:10.353 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.347+0000 7fd5e614d700 1 -- 172.21.15.88:0/1039333987 shutdown_connections 2023-10-18T16:12:10.353 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:10.347+0000 7fd5e614d700 1 -- 172.21.15.88:0/1039333987 wait complete. 2023-10-18T16:12:10.362 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:12:13.363 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:12:13.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.488+0000 7f34df5a7700 1 Processor -- start 2023-10-18T16:12:13.492 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.488+0000 7f34df5a7700 1 -- start start 2023-10-18T16:12:13.494 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.489+0000 7f34df5a7700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f34d80a9050 0x7f34d8153ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:13.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.489+0000 7f34df5a7700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34d8064150 con 0x7f34d8065360 2023-10-18T16:12:13.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.489+0000 7f34df5a7700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34d8153a40 con 0x7f34d814d8c0 2023-10-18T16:12:13.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.489+0000 7f34df5a7700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34d8063e70 con 0x7f34d80a9050 2023-10-18T16:12:13.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.489+0000 7f34dcb42700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f34d80a9050 0x7f34d8153ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:13.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.489+0000 7f34ddb44700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f34d8065360 0x7f34d815acb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:40368/0 (socket says 172.21.15.88:40368) 2023-10-18T16:12:13.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.489+0000 7f34ddb44700 1 -- 172.21.15.88:0/2819673599 learned_addr learned my addr 172.21.15.88:0/2819673599 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:12:13.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.489+0000 7f34dcb42700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f34d80a9050 0x7f34d8153ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41252/0 (socket says 172.21.15.88:41252) 2023-10-18T16:12:13.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.489+0000 7f34dcb42700 1 -- 172.21.15.88:0/2819673599 >> v1:172.21.15.151:6789/0 conn(0x7f34d814d8c0 legacy=0x7f34d8064440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:12:13.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.489+0000 7f34dcb42700 1 -- 172.21.15.88:0/2819673599 >> v1:172.21.15.88:6789/0 conn(0x7f34d8065360 legacy=0x7f34d815acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:13.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.489+0000 7f34cffff700 1 -- 172.21.15.88:0/2819673599 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3638578303 0 0) 0x7f34d8064150 con 0x7f34d8065360 2023-10-18T16:12:13.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.489+0000 7f34dcb42700 1 -- 172.21.15.88:0/2819673599 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34d80c72c0 con 0x7f34d80a9050 2023-10-18T16:12:13.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.489+0000 7f34dcb42700 1 --2- 172.21.15.88:0/2819673599 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34d80a9050 0x7f34d8153ed0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7f34c8009760 tx=0x7f34c8009060).ready entity=mon.0 client_cookie=cbd252378ae357b server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:12:13.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.489+0000 7f34cffff700 1 -- 172.21.15.88:0/2819673599 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34c80090d0 con 0x7f34d80a9050 2023-10-18T16:12:13.498 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.489+0000 7f34cffff700 1 -- 172.21.15.88:0/2819673599 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34c800b520 con 0x7f34d80a9050 2023-10-18T16:12:13.498 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.490+0000 7f34cffff700 1 -- 172.21.15.88:0/2819673599 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34c800b8a0 con 0x7f34d80a9050 2023-10-18T16:12:13.498 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.490+0000 7f34df5a7700 1 -- 172.21.15.88:0/2819673599 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34d80a9050 msgr2=0x7f34d8153ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:13.498 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.490+0000 7f34df5a7700 1 --2- 172.21.15.88:0/2819673599 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34d80a9050 0x7f34d8153ed0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7f34c8009760 tx=0x7f34c8009060).stop 2023-10-18T16:12:13.498 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.490+0000 7f34df5a7700 1 -- 172.21.15.88:0/2819673599 shutdown_connections 2023-10-18T16:12:13.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.490+0000 7f34df5a7700 1 --2- 172.21.15.88:0/2819673599 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34d80a9050 0x7f34d8153ed0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:13.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.490+0000 7f34df5a7700 1 -- 172.21.15.88:0/2819673599 >> 172.21.15.88:0/2819673599 conn(0x7f34d806a810 msgr2=0x7f34d80c6a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:12:13.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.490+0000 7f34df5a7700 1 -- 172.21.15.88:0/2819673599 shutdown_connections 2023-10-18T16:12:13.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.490+0000 7f34df5a7700 1 -- 172.21.15.88:0/2819673599 wait complete. 2023-10-18T16:12:13.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.491+0000 7f34df5a7700 1 Processor -- start 2023-10-18T16:12:13.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.491+0000 7f34df5a7700 1 -- start start 2023-10-18T16:12:13.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.492+0000 7f34df5a7700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f34d8065360 0x7f34d80bf7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:13.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.492+0000 7f34df5a7700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f34d80a9050 0x7f34d80bfc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:13.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.492+0000 7f34df5a7700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34d814d8c0 0x7f34d80c0570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:13.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.492+0000 7f34df5a7700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d80c0180 con 0x7f34d814d8c0 2023-10-18T16:12:13.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.492+0000 7f34df5a7700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d80bc900 con 0x7f34d80a9050 2023-10-18T16:12:13.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.492+0000 7f34df5a7700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34d80bcbe0 con 0x7f34d8065360 2023-10-18T16:12:13.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.492+0000 7f34ddb44700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34d814d8c0 0x7f34d80c0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:13.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.492+0000 7f34ddb44700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34d814d8c0 0x7f34d80c0570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41254/0 (socket says 172.21.15.88:41254) 2023-10-18T16:12:13.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.492+0000 7f34ddb44700 1 -- 172.21.15.88:0/456085465 learned_addr learned my addr 172.21.15.88:0/456085465 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:12:13.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.492+0000 7f34dcb42700 1 --2- 172.21.15.88:0/456085465 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f34d80a9050 0x7f34d80bfc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:13.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.492+0000 7f34dd343700 1 --2- 172.21.15.88:0/456085465 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f34d8065360 0x7f34d80bf7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:13.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.493+0000 7f34ddb44700 1 -- 172.21.15.88:0/456085465 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f34d8065360 msgr2=0x7f34d80bf7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:13.505 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.493+0000 7f34ddb44700 1 --2- 172.21.15.88:0/456085465 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f34d8065360 0x7f34d80bf7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:13.505 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.493+0000 7f34ddb44700 1 -- 172.21.15.88:0/456085465 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f34d80a9050 msgr2=0x7f34d80bfc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:13.505 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.493+0000 7f34ddb44700 1 --2- 172.21.15.88:0/456085465 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f34d80a9050 0x7f34d80bfc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:13.505 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.493+0000 7f34ddb44700 1 -- 172.21.15.88:0/456085465 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34c80092a0 con 0x7f34d814d8c0 2023-10-18T16:12:13.505 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.493+0000 7f34dcb42700 1 --2- 172.21.15.88:0/456085465 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f34d80a9050 0x7f34d80bfc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-10-18T16:12:13.506 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.493+0000 7f34dd343700 1 --2- 172.21.15.88:0/456085465 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f34d8065360 0x7f34d80bf7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-10-18T16:12:13.506 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.493+0000 7f34ddb44700 1 --2- 172.21.15.88:0/456085465 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34d814d8c0 0x7f34d80c0570 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f34d000c790 tx=0x7f34d0007f30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:12:13.506 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.493+0000 7f34cd7fa700 1 -- 172.21.15.88:0/456085465 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34d0005d30 con 0x7f34d814d8c0 2023-10-18T16:12:13.506 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.493+0000 7f34cd7fa700 1 -- 172.21.15.88:0/456085465 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34d0007110 con 0x7f34d814d8c0 2023-10-18T16:12:13.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.493+0000 7f34df5a7700 1 -- 172.21.15.88:0/456085465 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34d80bcec0 con 0x7f34d814d8c0 2023-10-18T16:12:13.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.493+0000 7f34cd7fa700 1 -- 172.21.15.88:0/456085465 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34d00074d0 con 0x7f34d814d8c0 2023-10-18T16:12:13.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.493+0000 7f34df5a7700 1 -- 172.21.15.88:0/456085465 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34d80bd480 con 0x7f34d814d8c0 2023-10-18T16:12:13.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.494+0000 7f34df5a7700 1 -- 172.21.15.88:0/456085465 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34d806c4e0 con 0x7f34d814d8c0 2023-10-18T16:12:13.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.494+0000 7f34cd7fa700 1 -- 172.21.15.88:0/456085465 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f34d0007840 con 0x7f34d814d8c0 2023-10-18T16:12:13.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.495+0000 7f34cd7fa700 1 --2- 172.21.15.88:0/456085465 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f34bc05f120 0x7f34bc0615e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:13.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.495+0000 7f34cd7fa700 1 -- 172.21.15.88:0/456085465 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6725+0+0 (secure 0 0 0) 0x7f34d007a4f0 con 0x7f34d814d8c0 2023-10-18T16:12:13.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.495+0000 7f34dd343700 1 --2- 172.21.15.88:0/456085465 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f34bc05f120 0x7f34bc0615e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:13.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.495+0000 7f34dd343700 1 --2- 172.21.15.88:0/456085465 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f34bc05f120 0x7f34bc0615e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f34d400a820 tx=0x7f34d4008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:12:13.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.499+0000 7f34cd7fa700 1 -- 172.21.15.88:0/456085465 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f34d004b560 con 0x7f34d814d8c0 2023-10-18T16:12:13.667 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~158s 2023-10-18T16:12:13.667 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~171s 2023-10-18T16:12:13.667 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~203s 2023-10-18T16:12:13.667 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~184s 2023-10-18T16:12:13.668 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~265s 2023-10-18T16:12:13.668 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.f is failed for ~217s 2023-10-18T16:12:13.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.703+0000 7f34df5a7700 1 -- 172.21.15.88:0/456085465 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f34d80c75a0 con 0x7f34d814d8c0 2023-10-18T16:12:13.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.703+0000 7f34cd7fa700 1 -- 172.21.15.88:0/456085465 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7f34d004eb80 con 0x7f34d814d8c0 2023-10-18T16:12:13.707 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:12:13.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.705+0000 7f34df5a7700 1 -- 172.21.15.88:0/456085465 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f34bc05f120 msgr2=0x7f34bc0615e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:13.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.705+0000 7f34df5a7700 1 --2- 172.21.15.88:0/456085465 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f34bc05f120 0x7f34bc0615e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f34d400a820 tx=0x7f34d4008040).stop 2023-10-18T16:12:13.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.705+0000 7f34df5a7700 1 -- 172.21.15.88:0/456085465 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34d814d8c0 msgr2=0x7f34d80c0570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:13.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.705+0000 7f34df5a7700 1 --2- 172.21.15.88:0/456085465 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34d814d8c0 0x7f34d80c0570 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f34d000c790 tx=0x7f34d0007f30).stop 2023-10-18T16:12:13.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.705+0000 7f34df5a7700 1 -- 172.21.15.88:0/456085465 shutdown_connections 2023-10-18T16:12:13.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.705+0000 7f34df5a7700 1 --2- 172.21.15.88:0/456085465 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f34bc05f120 0x7f34bc0615e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:13.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.705+0000 7f34df5a7700 1 --2- 172.21.15.88:0/456085465 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34d814d8c0 0x7f34d80c0570 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:13.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.705+0000 7f34df5a7700 1 --2- 172.21.15.88:0/456085465 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f34d8065360 0x7f34d80bf7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:13.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.705+0000 7f34df5a7700 1 --2- 172.21.15.88:0/456085465 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f34d80a9050 0x7f34d80bfc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:13.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.705+0000 7f34df5a7700 1 -- 172.21.15.88:0/456085465 >> 172.21.15.88:0/456085465 conn(0x7f34d806a810 msgr2=0x7f34d80c19b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:12:13.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.705+0000 7f34df5a7700 1 -- 172.21.15.88:0/456085465 shutdown_connections 2023-10-18T16:12:13.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:13.705+0000 7f34df5a7700 1 -- 172.21.15.88:0/456085465 wait complete. 2023-10-18T16:12:13.720 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:12:16.721 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:12:16.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:16.843+0000 7f0173651700 1 Processor -- start 2023-10-18T16:12:16.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:16.843+0000 7f0173651700 1 -- start start 2023-10-18T16:12:16.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:16.844+0000 7f0173651700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f016c1518d0 0x7f016c0bb650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:16.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:16.844+0000 7f0173651700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f016c0b9e10 con 0x7f016c0bf230 2023-10-18T16:12:16.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:16.844+0000 7f0173651700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f016c068ea0 con 0x7f016c066b00 2023-10-18T16:12:16.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:16.844+0000 7f0173651700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f016c0bf770 con 0x7f016c1518d0 2023-10-18T16:12:16.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:16.844+0000 7f01713ed700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f016c1518d0 0x7f016c0bb650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:16.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:16.844+0000 7f01713ed700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f016c1518d0 0x7f016c0bb650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41262/0 (socket says 172.21.15.88:41262) 2023-10-18T16:12:16.850 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:16.844+0000 7f01713ed700 1 -- 172.21.15.88:0/2142774745 learned_addr learned my addr 172.21.15.88:0/2142774745 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:12:16.850 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:16.844+0000 7f01713ed700 1 -- 172.21.15.88:0/2142774745 >> v1:172.21.15.151:6789/0 conn(0x7f016c066b00 legacy=0x7f016c066f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:12:16.850 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:16.844+0000 7f01713ed700 1 -- 172.21.15.88:0/2142774745 >> v1:172.21.15.88:6789/0 conn(0x7f016c0bf230 legacy=0x7f016c151cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:16.850 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:16.844+0000 7f0163fff700 1 -- 172.21.15.88:0/2142774745 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2212504774 0 0) 0x7f016c0b9e10 con 0x7f016c0bf230 2023-10-18T16:12:16.850 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:16.844+0000 7f01713ed700 1 -- 172.21.15.88:0/2142774745 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f016c069150 con 0x7f016c1518d0 2023-10-18T16:12:16.851 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:16.844+0000 7f01713ed700 1 --2- 172.21.15.88:0/2142774745 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f016c1518d0 0x7f016c0bb650 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f0168009f20 tx=0x7f0168009840).ready entity=mon.0 client_cookie=a885ca252d2017bf server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:12:16.851 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:16.844+0000 7f0163fff700 1 -- 172.21.15.88:0/2142774745 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f016800e070 con 0x7f016c1518d0 2023-10-18T16:12:16.851 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:16.844+0000 7f0163fff700 1 -- 172.21.15.88:0/2142774745 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f016800ba80 con 0x7f016c1518d0 2023-10-18T16:12:16.851 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:16.844+0000 7f0163fff700 1 -- 172.21.15.88:0/2142774745 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f016800be00 con 0x7f016c1518d0 2023-10-18T16:12:16.851 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:16.845+0000 7f0173651700 1 -- 172.21.15.88:0/2142774745 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f016c1518d0 msgr2=0x7f016c0bb650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:16.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:16.845+0000 7f0173651700 1 --2- 172.21.15.88:0/2142774745 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f016c1518d0 0x7f016c0bb650 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f0168009f20 tx=0x7f0168009840).stop 2023-10-18T16:12:16.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:16.845+0000 7f0173651700 1 -- 172.21.15.88:0/2142774745 shutdown_connections 2023-10-18T16:12:16.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:16.845+0000 7f0173651700 1 --2- 172.21.15.88:0/2142774745 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f016c1518d0 0x7f016c0bb650 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:16.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:16.845+0000 7f0173651700 1 -- 172.21.15.88:0/2142774745 >> 172.21.15.88:0/2142774745 conn(0x7f016c0c4890 msgr2=0x7f016c0c4ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:12:16.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:16.845+0000 7f0173651700 1 -- 172.21.15.88:0/2142774745 shutdown_connections 2023-10-18T16:12:16.853 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:16.845+0000 7f0173651700 1 -- 172.21.15.88:0/2142774745 wait complete. 2023-10-18T16:12:16.855 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:16.846+0000 7f0173651700 1 Processor -- start 2023-10-18T16:12:16.856 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:16.846+0000 7f0173651700 1 -- start start 2023-10-18T16:12:16.856 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:16.847+0000 7f0173651700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f016c066b00 0x7f016c0a1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:16.856 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:16.847+0000 7f0173651700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f016c0bf230 0x7f016c161590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:16.856 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:16.847+0000 7f0173651700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f016c1518d0 0x7f016c0c2d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:16.856 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:16.847+0000 7f0173651700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f016c15e830 con 0x7f016c1518d0 2023-10-18T16:12:16.857 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:16.847+0000 7f0173651700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f016c162400 con 0x7f016c066b00 2023-10-18T16:12:16.857 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:16.847+0000 7f0173651700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f016c161e40 con 0x7f016c0bf230 2023-10-18T16:12:16.857 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:16.847+0000 7f0171bee700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f016c1518d0 0x7f016c0c2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:16.857 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:16.847+0000 7f0171bee700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f016c1518d0 0x7f016c0c2d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41266/0 (socket says 172.21.15.88:41266) 2023-10-18T16:12:16.857 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:16.847+0000 7f0171bee700 1 -- 172.21.15.88:0/2835299472 learned_addr learned my addr 172.21.15.88:0/2835299472 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:12:16.858 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:16.847+0000 7f01713ed700 1 --2- 172.21.15.88:0/2835299472 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f016c066b00 0x7f016c0a1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:16.858 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:16.847+0000 7f0170bec700 1 --2- 172.21.15.88:0/2835299472 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f016c0bf230 0x7f016c161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:16.858 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:16.847+0000 7f0171bee700 1 -- 172.21.15.88:0/2835299472 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f016c0bf230 msgr2=0x7f016c161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:16.858 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:16.847+0000 7f0171bee700 1 --2- 172.21.15.88:0/2835299472 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f016c0bf230 0x7f016c161590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:16.858 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:16.847+0000 7f0171bee700 1 -- 172.21.15.88:0/2835299472 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f016c066b00 msgr2=0x7f016c0a1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:16.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:16.847+0000 7f0171bee700 1 --2- 172.21.15.88:0/2835299472 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f016c066b00 0x7f016c0a1720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:16.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:16.847+0000 7f0171bee700 1 -- 172.21.15.88:0/2835299472 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0168009ae0 con 0x7f016c1518d0 2023-10-18T16:12:16.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:16.847+0000 7f01713ed700 1 --2- 172.21.15.88:0/2835299472 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f016c066b00 0x7f016c0a1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:12:16.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:16.848+0000 7f0171bee700 1 --2- 172.21.15.88:0/2835299472 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f016c1518d0 0x7f016c0c2d80 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7f016400c490 tx=0x7f0164007de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:12:16.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:16.848+0000 7f01617fa700 1 -- 172.21.15.88:0/2835299472 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0164002ea0 con 0x7f016c1518d0 2023-10-18T16:12:16.860 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:16.848+0000 7f01617fa700 1 -- 172.21.15.88:0/2835299472 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f016400f470 con 0x7f016c1518d0 2023-10-18T16:12:16.860 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:16.848+0000 7f0173651700 1 -- 172.21.15.88:0/2835299472 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f016c162120 con 0x7f016c1518d0 2023-10-18T16:12:16.860 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:16.848+0000 7f01617fa700 1 -- 172.21.15.88:0/2835299472 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f016400f810 con 0x7f016c1518d0 2023-10-18T16:12:16.860 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:16.848+0000 7f0173651700 1 -- 172.21.15.88:0/2835299472 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f016c15e4c0 con 0x7f016c1518d0 2023-10-18T16:12:16.860 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:16.849+0000 7f0173651700 1 -- 172.21.15.88:0/2835299472 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f016c15e040 con 0x7f016c1518d0 2023-10-18T16:12:16.861 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:16.849+0000 7f01617fa700 1 -- 172.21.15.88:0/2835299472 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f016400fc40 con 0x7f016c1518d0 2023-10-18T16:12:16.861 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:16.849+0000 7f01617fa700 1 --2- 172.21.15.88:0/2835299472 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f015005f180 0x7f0150061640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:16.861 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:16.849+0000 7f01617fa700 1 -- 172.21.15.88:0/2835299472 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6725+0+0 (secure 0 0 0) 0x7f0164079800 con 0x7f016c1518d0 2023-10-18T16:12:16.861 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:16.849+0000 7f01713ed700 1 --2- 172.21.15.88:0/2835299472 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f015005f180 0x7f0150061640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:16.861 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:16.850+0000 7f01713ed700 1 --2- 172.21.15.88:0/2835299472 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f015005f180 0x7f0150061640 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f0168009880 tx=0x7f016c0bb000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:12:16.862 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:16.853+0000 7f01617fa700 1 -- 172.21.15.88:0/2835299472 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f0164047ac0 con 0x7f016c1518d0 2023-10-18T16:12:17.055 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:17.051+0000 7f0173651700 1 -- 172.21.15.88:0/2835299472 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f016c15e9e0 con 0x7f016c1518d0 2023-10-18T16:12:17.055 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:17.051+0000 7f01617fa700 1 -- 172.21.15.88:0/2835299472 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7f016c15e9e0 con 0x7f016c1518d0 2023-10-18T16:12:17.056 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:12:17.057 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:17.053+0000 7f0173651700 1 -- 172.21.15.88:0/2835299472 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f015005f180 msgr2=0x7f0150061640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:17.057 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:17.053+0000 7f0173651700 1 --2- 172.21.15.88:0/2835299472 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f015005f180 0x7f0150061640 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f0168009880 tx=0x7f016c0bb000).stop 2023-10-18T16:12:17.057 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:17.053+0000 7f0173651700 1 -- 172.21.15.88:0/2835299472 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f016c1518d0 msgr2=0x7f016c0c2d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:17.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:17.053+0000 7f0173651700 1 --2- 172.21.15.88:0/2835299472 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f016c1518d0 0x7f016c0c2d80 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7f016400c490 tx=0x7f0164007de0).stop 2023-10-18T16:12:17.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:17.053+0000 7f0173651700 1 -- 172.21.15.88:0/2835299472 shutdown_connections 2023-10-18T16:12:17.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:17.053+0000 7f0173651700 1 --2- 172.21.15.88:0/2835299472 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f015005f180 0x7f0150061640 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:17.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:17.053+0000 7f0173651700 1 --2- 172.21.15.88:0/2835299472 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f016c1518d0 0x7f016c0c2d80 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:17.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:17.053+0000 7f0173651700 1 --2- 172.21.15.88:0/2835299472 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f016c066b00 0x7f016c0a1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:17.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:17.053+0000 7f0173651700 1 --2- 172.21.15.88:0/2835299472 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f016c0bf230 0x7f016c161590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:17.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:17.053+0000 7f0173651700 1 -- 172.21.15.88:0/2835299472 >> 172.21.15.88:0/2835299472 conn(0x7f016c0c4890 msgr2=0x7f016c14b630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:12:17.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:17.053+0000 7f0173651700 1 -- 172.21.15.88:0/2835299472 shutdown_connections 2023-10-18T16:12:17.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:17.053+0000 7f0173651700 1 -- 172.21.15.88:0/2835299472 wait complete. 2023-10-18T16:12:17.068 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:12:19.978 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~164s 2023-10-18T16:12:19.978 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~177s 2023-10-18T16:12:19.978 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~210s 2023-10-18T16:12:19.978 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~190s 2023-10-18T16:12:19.979 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~271s 2023-10-18T16:12:19.979 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.f is failed for ~223s 2023-10-18T16:12:20.070 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:12:20.191 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.187+0000 7ff1dea3c700 1 Processor -- start 2023-10-18T16:12:20.191 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.187+0000 7ff1dea3c700 1 -- start start 2023-10-18T16:12:20.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.188+0000 7ff1dea3c700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff1d80a6e80 0x7ff1d8064ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:20.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.188+0000 7ff1dea3c700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1d8065390 con 0x7ff1d814d8f0 2023-10-18T16:12:20.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.188+0000 7ff1dea3c700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1d80a87f0 con 0x7ff1d8152510 2023-10-18T16:12:20.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.188+0000 7ff1dea3c700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1d80a8ad0 con 0x7ff1d80a6e80 2023-10-18T16:12:20.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.188+0000 7ff1d7fff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff1d80a6e80 0x7ff1d8064ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:20.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.188+0000 7ff1d77fe700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7ff1d814d8f0 0x7ff1d8151e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:44926/0 (socket says 172.21.15.88:44926) 2023-10-18T16:12:20.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.188+0000 7ff1d7fff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff1d80a6e80 0x7ff1d8064ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:34964/0 (socket says 172.21.15.88:34964) 2023-10-18T16:12:20.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.188+0000 7ff1d77fe700 1 -- 172.21.15.88:0/3764737451 learned_addr learned my addr 172.21.15.88:0/3764737451 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:12:20.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.188+0000 7ff1d7fff700 1 -- 172.21.15.88:0/3764737451 >> v1:172.21.15.151:6789/0 conn(0x7ff1d8152510 legacy=0x7ff1d8150640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:20.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.188+0000 7ff1d7fff700 1 -- 172.21.15.88:0/3764737451 >> v1:172.21.15.88:6789/0 conn(0x7ff1d814d8f0 legacy=0x7ff1d8151e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:20.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.188+0000 7ff1d7fff700 1 -- 172.21.15.88:0/3764737451 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1d80a8db0 con 0x7ff1d80a6e80 2023-10-18T16:12:20.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.188+0000 7ff1d6ffd700 1 -- 172.21.15.88:0/3764737451 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1357476043 0 0) 0x7ff1d8065390 con 0x7ff1d814d8f0 2023-10-18T16:12:20.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.189+0000 7ff1d7fff700 1 --2- 172.21.15.88:0/3764737451 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1d80a6e80 0x7ff1d8064ea0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7ff1c8009760 tx=0x7ff1c8009060).ready entity=mon.0 client_cookie=373c01132be60618 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:12:20.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.189+0000 7ff1d6ffd700 1 -- 172.21.15.88:0/3764737451 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1c8009120 con 0x7ff1d80a6e80 2023-10-18T16:12:20.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.189+0000 7ff1d6ffd700 1 -- 172.21.15.88:0/3764737451 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1c800b580 con 0x7ff1d80a6e80 2023-10-18T16:12:20.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.189+0000 7ff1d6ffd700 1 -- 172.21.15.88:0/3764737451 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1c800b900 con 0x7ff1d80a6e80 2023-10-18T16:12:20.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.189+0000 7ff1dea3c700 1 -- 172.21.15.88:0/3764737451 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1d80a6e80 msgr2=0x7ff1d8064ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:20.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.189+0000 7ff1dea3c700 1 --2- 172.21.15.88:0/3764737451 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1d80a6e80 0x7ff1d8064ea0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7ff1c8009760 tx=0x7ff1c8009060).stop 2023-10-18T16:12:20.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.189+0000 7ff1dea3c700 1 -- 172.21.15.88:0/3764737451 shutdown_connections 2023-10-18T16:12:20.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.189+0000 7ff1dea3c700 1 --2- 172.21.15.88:0/3764737451 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1d80a6e80 0x7ff1d8064ea0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:20.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.189+0000 7ff1dea3c700 1 -- 172.21.15.88:0/3764737451 >> 172.21.15.88:0/3764737451 conn(0x7ff1d8069d80 msgr2=0x7ff1d806a190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:12:20.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.189+0000 7ff1dea3c700 1 -- 172.21.15.88:0/3764737451 shutdown_connections 2023-10-18T16:12:20.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.189+0000 7ff1dea3c700 1 -- 172.21.15.88:0/3764737451 wait complete. 2023-10-18T16:12:20.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.190+0000 7ff1dea3c700 1 Processor -- start 2023-10-18T16:12:20.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.191+0000 7ff1dea3c700 1 -- start start 2023-10-18T16:12:20.201 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.191+0000 7ff1dea3c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff1d80a6e80 0x7ff1d80aa340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:20.201 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.191+0000 7ff1dea3c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1d814d8f0 0x7ff1d80c4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:20.201 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.191+0000 7ff1dea3c700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff1d8152510 0x7ff1d814e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:20.201 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.191+0000 7ff1dea3c700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d81509e0 con 0x7ff1d814d8f0 2023-10-18T16:12:20.201 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.191+0000 7ff1dea3c700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d8164110 con 0x7ff1d80a6e80 2023-10-18T16:12:20.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.191+0000 7ff1dea3c700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d8156530 con 0x7ff1d8152510 2023-10-18T16:12:20.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.191+0000 7ff1d77fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1d814d8f0 0x7ff1d80c4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:20.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.191+0000 7ff1d77fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1d814d8f0 0x7ff1d80c4a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:34980/0 (socket says 172.21.15.88:34980) 2023-10-18T16:12:20.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.191+0000 7ff1d77fe700 1 -- 172.21.15.88:0/3764753760 learned_addr learned my addr 172.21.15.88:0/3764753760 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:12:20.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.192+0000 7ff1d7fff700 1 --2- 172.21.15.88:0/3764753760 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff1d80a6e80 0x7ff1d80aa340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:20.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.192+0000 7ff1dcfd9700 1 --2- 172.21.15.88:0/3764753760 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff1d8152510 0x7ff1d814e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:20.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.192+0000 7ff1d77fe700 1 -- 172.21.15.88:0/3764753760 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff1d8152510 msgr2=0x7ff1d814e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:20.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.192+0000 7ff1d77fe700 1 --2- 172.21.15.88:0/3764753760 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff1d8152510 0x7ff1d814e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:20.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.192+0000 7ff1d77fe700 1 -- 172.21.15.88:0/3764753760 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff1d80a6e80 msgr2=0x7ff1d80aa340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:20.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.192+0000 7ff1d77fe700 1 --2- 172.21.15.88:0/3764753760 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff1d80a6e80 0x7ff1d80aa340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:20.204 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.192+0000 7ff1d77fe700 1 -- 172.21.15.88:0/3764753760 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1c80092a0 con 0x7ff1d814d8f0 2023-10-18T16:12:20.204 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.192+0000 7ff1dcfd9700 1 --2- 172.21.15.88:0/3764753760 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff1d8152510 0x7ff1d814e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-10-18T16:12:20.204 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.192+0000 7ff1d77fe700 1 --2- 172.21.15.88:0/3764753760 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1d814d8f0 0x7ff1d80c4a80 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7ff1cc00c4f0 tx=0x7ff1cc007c10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:12:20.204 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.192+0000 7ff1bffff700 1 -- 172.21.15.88:0/3764753760 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1cc0072e0 con 0x7ff1d814d8f0 2023-10-18T16:12:20.204 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.192+0000 7ff1bffff700 1 -- 172.21.15.88:0/3764753760 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1cc007440 con 0x7ff1d814d8f0 2023-10-18T16:12:20.205 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.192+0000 7ff1bffff700 1 -- 172.21.15.88:0/3764753760 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1cc0057a0 con 0x7ff1d814d8f0 2023-10-18T16:12:20.205 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.192+0000 7ff1dea3c700 1 -- 172.21.15.88:0/3764753760 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1d81634d0 con 0x7ff1d814d8f0 2023-10-18T16:12:20.205 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.192+0000 7ff1dea3c700 1 -- 172.21.15.88:0/3764753760 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1d8163a50 con 0x7ff1d814d8f0 2023-10-18T16:12:20.205 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.193+0000 7ff1dea3c700 1 -- 172.21.15.88:0/3764753760 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1d8164250 con 0x7ff1d814d8f0 2023-10-18T16:12:20.205 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.193+0000 7ff1bffff700 1 -- 172.21.15.88:0/3764753760 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7ff1cc005b90 con 0x7ff1d814d8f0 2023-10-18T16:12:20.205 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.193+0000 7ff1bffff700 1 --2- 172.21.15.88:0/3764753760 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7ff1c405f150 0x7ff1c4061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:20.206 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.194+0000 7ff1bffff700 1 -- 172.21.15.88:0/3764753760 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6725+0+0 (secure 0 0 0) 0x7ff1cc079c30 con 0x7ff1d814d8f0 2023-10-18T16:12:20.206 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.194+0000 7ff1d7fff700 1 --2- 172.21.15.88:0/3764753760 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7ff1c405f150 0x7ff1c4061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:20.206 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.194+0000 7ff1d7fff700 1 --2- 172.21.15.88:0/3764753760 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7ff1c405f150 0x7ff1c4061610 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7ff1c800b340 tx=0x7ff1c8017000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:12:20.207 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.198+0000 7ff1bffff700 1 -- 172.21.15.88:0/3764753760 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7ff1cc04aca0 con 0x7ff1d814d8f0 2023-10-18T16:12:20.404 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.400+0000 7ff1dea3c700 1 -- 172.21.15.88:0/3764753760 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff1d806c4e0 con 0x7ff1d814d8f0 2023-10-18T16:12:20.404 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.400+0000 7ff1bffff700 1 -- 172.21.15.88:0/3764753760 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7ff1cc079030 con 0x7ff1d814d8f0 2023-10-18T16:12:20.404 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:12:20.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.402+0000 7ff1dea3c700 1 -- 172.21.15.88:0/3764753760 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7ff1c405f150 msgr2=0x7ff1c4061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:20.407 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.402+0000 7ff1dea3c700 1 --2- 172.21.15.88:0/3764753760 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7ff1c405f150 0x7ff1c4061610 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7ff1c800b340 tx=0x7ff1c8017000).stop 2023-10-18T16:12:20.407 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.402+0000 7ff1dea3c700 1 -- 172.21.15.88:0/3764753760 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1d814d8f0 msgr2=0x7ff1d80c4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:20.407 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.402+0000 7ff1dea3c700 1 --2- 172.21.15.88:0/3764753760 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1d814d8f0 0x7ff1d80c4a80 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7ff1cc00c4f0 tx=0x7ff1cc007c10).stop 2023-10-18T16:12:20.407 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.402+0000 7ff1dea3c700 1 -- 172.21.15.88:0/3764753760 shutdown_connections 2023-10-18T16:12:20.407 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.402+0000 7ff1dea3c700 1 --2- 172.21.15.88:0/3764753760 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7ff1c405f150 0x7ff1c4061610 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:20.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.402+0000 7ff1dea3c700 1 --2- 172.21.15.88:0/3764753760 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff1d8152510 0x7ff1d814e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:20.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.402+0000 7ff1dea3c700 1 --2- 172.21.15.88:0/3764753760 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff1d80a6e80 0x7ff1d80aa340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:20.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.402+0000 7ff1dea3c700 1 --2- 172.21.15.88:0/3764753760 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1d814d8f0 0x7ff1d80c4a80 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:20.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.402+0000 7ff1dea3c700 1 -- 172.21.15.88:0/3764753760 >> 172.21.15.88:0/3764753760 conn(0x7ff1d8069d80 msgr2=0x7ff1d8064620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:12:20.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.402+0000 7ff1dea3c700 1 -- 172.21.15.88:0/3764753760 shutdown_connections 2023-10-18T16:12:20.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:20.402+0000 7ff1dea3c700 1 -- 172.21.15.88:0/3764753760 wait complete. 2023-10-18T16:12:20.417 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:12:23.418 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:12:23.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.539+0000 7f19561bf700 1 Processor -- start 2023-10-18T16:12:23.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.539+0000 7f19561bf700 1 -- start start 2023-10-18T16:12:23.546 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.540+0000 7f19561bf700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f1950153010 0x7f19501533f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:23.546 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.540+0000 7f19561bf700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f19501525f0 0x7f19500c36c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:23.547 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.540+0000 7f19561bf700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19500a9670 con 0x7f1950064d70 2023-10-18T16:12:23.547 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.540+0000 7f19561bf700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f195014df50 con 0x7f19501525f0 2023-10-18T16:12:23.547 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.540+0000 7f19561bf700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1950151a50 con 0x7f1950153010 2023-10-18T16:12:23.547 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.540+0000 7f194ffff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f19501525f0 0x7f19500c36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:23.547 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.540+0000 7f194effd700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f1950064d70 0x7f1950151ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:44930/0 (socket says 172.21.15.88:44930) 2023-10-18T16:12:23.548 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.540+0000 7f194effd700 1 -- 172.21.15.88:0/1079015218 learned_addr learned my addr 172.21.15.88:0/1079015218 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:12:23.548 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.540+0000 7f194ffff700 1 --2- 172.21.15.88:0/1079015218 >> v2:172.21.15.88:3300/0 conn(0x7f19501525f0 0x7f19500c36c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:34994/0 (socket says 172.21.15.88:34994) 2023-10-18T16:12:23.548 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.540+0000 7f194f7fe700 1 --2- 172.21.15.88:0/1079015218 >> v2:172.21.15.151:3300/0 conn(0x7f1950153010 0x7f19501533f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:23.548 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.541+0000 7f194ffff700 1 -- 172.21.15.88:0/1079015218 >> v2:172.21.15.151:3300/0 conn(0x7f1950153010 msgr2=0x7f19501533f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:12:23.548 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.541+0000 7f194ffff700 1 --2- 172.21.15.88:0/1079015218 >> v2:172.21.15.151:3300/0 conn(0x7f1950153010 0x7f19501533f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:23.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.541+0000 7f194ffff700 1 -- 172.21.15.88:0/1079015218 >> v1:172.21.15.88:6789/0 conn(0x7f1950064d70 legacy=0x7f1950151ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:23.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.541+0000 7f194e7fc700 1 -- 172.21.15.88:0/1079015218 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4003455703 0 0) 0x7f19500a9670 con 0x7f1950064d70 2023-10-18T16:12:23.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.541+0000 7f194ffff700 1 -- 172.21.15.88:0/1079015218 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19500a8af0 con 0x7f19501525f0 2023-10-18T16:12:23.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.541+0000 7f194ffff700 1 --2- 172.21.15.88:0/1079015218 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f19501525f0 0x7f19500c36c0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f195006aaa0 tx=0x7f1948009060).ready entity=mon.0 client_cookie=a45a8f02d95e53ff server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:12:23.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.541+0000 7f194e7fc700 1 -- 172.21.15.88:0/1079015218 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19480090d0 con 0x7f19501525f0 2023-10-18T16:12:23.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.541+0000 7f194e7fc700 1 -- 172.21.15.88:0/1079015218 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f194800b580 con 0x7f19501525f0 2023-10-18T16:12:23.550 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.541+0000 7f194e7fc700 1 -- 172.21.15.88:0/1079015218 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f194800b900 con 0x7f19501525f0 2023-10-18T16:12:23.550 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.541+0000 7f19561bf700 1 -- 172.21.15.88:0/1079015218 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f19501525f0 msgr2=0x7f19500c36c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:23.550 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.541+0000 7f19561bf700 1 --2- 172.21.15.88:0/1079015218 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f19501525f0 0x7f19500c36c0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f195006aaa0 tx=0x7f1948009060).stop 2023-10-18T16:12:23.550 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.541+0000 7f19561bf700 1 -- 172.21.15.88:0/1079015218 shutdown_connections 2023-10-18T16:12:23.550 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.541+0000 7f19561bf700 1 --2- 172.21.15.88:0/1079015218 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f19501525f0 0x7f19500c36c0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:23.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.541+0000 7f19561bf700 1 --2- 172.21.15.88:0/1079015218 >> v2:172.21.15.151:3300/0 conn(0x7f1950153010 0x7f19501533f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:23.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.541+0000 7f19561bf700 1 -- 172.21.15.88:0/1079015218 >> 172.21.15.88:0/1079015218 conn(0x7f1950069d80 msgr2=0x7f195006a190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:12:23.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.541+0000 7f19561bf700 1 -- 172.21.15.88:0/1079015218 shutdown_connections 2023-10-18T16:12:23.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.541+0000 7f19561bf700 1 -- 172.21.15.88:0/1079015218 wait complete. 2023-10-18T16:12:23.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.543+0000 7f19561bf700 1 Processor -- start 2023-10-18T16:12:23.552 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.543+0000 7f19561bf700 1 -- start start 2023-10-18T16:12:23.554 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.543+0000 7f19561bf700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1950064d70 0x7f195015e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:23.554 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.543+0000 7f19561bf700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f19501525f0 0x7f195015e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:23.554 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.543+0000 7f19561bf700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1950153010 0x7f1950163900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:23.555 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.543+0000 7f19561bf700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19500c7750 con 0x7f1950153010 2023-10-18T16:12:23.555 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.543+0000 7f19561bf700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f195014e2e0 con 0x7f1950064d70 2023-10-18T16:12:23.555 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.543+0000 7f19561bf700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19500c72c0 con 0x7f19501525f0 2023-10-18T16:12:23.555 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.544+0000 7f194ffff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1950153010 0x7f1950163900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:23.555 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.544+0000 7f194ffff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1950153010 0x7f1950163900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35004/0 (socket says 172.21.15.88:35004) 2023-10-18T16:12:23.556 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.544+0000 7f194ffff700 1 -- 172.21.15.88:0/475126070 learned_addr learned my addr 172.21.15.88:0/475126070 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:12:23.556 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.544+0000 7f194f7fe700 1 --2- 172.21.15.88:0/475126070 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1950064d70 0x7f195015e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:23.556 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.544+0000 7f194effd700 1 --2- 172.21.15.88:0/475126070 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f19501525f0 0x7f195015e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:23.556 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.544+0000 7f194ffff700 1 -- 172.21.15.88:0/475126070 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f19501525f0 msgr2=0x7f195015e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:23.556 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.544+0000 7f194ffff700 1 --2- 172.21.15.88:0/475126070 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f19501525f0 0x7f195015e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:23.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.544+0000 7f194ffff700 1 -- 172.21.15.88:0/475126070 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1950064d70 msgr2=0x7f195015e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:23.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.544+0000 7f194ffff700 1 --2- 172.21.15.88:0/475126070 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1950064d70 0x7f195015e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:23.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.544+0000 7f194ffff700 1 -- 172.21.15.88:0/475126070 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19480092a0 con 0x7f1950153010 2023-10-18T16:12:23.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.544+0000 7f194ffff700 1 --2- 172.21.15.88:0/475126070 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1950153010 0x7f1950163900 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7f1950084410 tx=0x7f1948017000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:12:23.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.544+0000 7f19377fe700 1 -- 172.21.15.88:0/475126070 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f194800b900 con 0x7f1950153010 2023-10-18T16:12:23.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.544+0000 7f19377fe700 1 -- 172.21.15.88:0/475126070 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1948006950 con 0x7f1950153010 2023-10-18T16:12:23.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.544+0000 7f19561bf700 1 -- 172.21.15.88:0/475126070 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19500a3660 con 0x7f1950153010 2023-10-18T16:12:23.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.544+0000 7f19377fe700 1 -- 172.21.15.88:0/475126070 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f194801b410 con 0x7f1950153010 2023-10-18T16:12:23.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.544+0000 7f19561bf700 1 -- 172.21.15.88:0/475126070 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19500c69b0 con 0x7f1950153010 2023-10-18T16:12:23.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.545+0000 7f19561bf700 1 -- 172.21.15.88:0/475126070 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19500c7470 con 0x7f1950153010 2023-10-18T16:12:23.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.546+0000 7f19377fe700 1 -- 172.21.15.88:0/475126070 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f194801b7c0 con 0x7f1950153010 2023-10-18T16:12:23.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.546+0000 7f19377fe700 1 --2- 172.21.15.88:0/475126070 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f193c05f150 0x7f193c061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:23.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.546+0000 7f19377fe700 1 -- 172.21.15.88:0/475126070 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6725+0+0 (secure 0 0 0) 0x7f194807be90 con 0x7f1950153010 2023-10-18T16:12:23.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.546+0000 7f194f7fe700 1 --2- 172.21.15.88:0/475126070 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f193c05f150 0x7f193c061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:23.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.547+0000 7f194f7fe700 1 --2- 172.21.15.88:0/475126070 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f193c05f150 0x7f193c061610 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f19400075a0 tx=0x7f19400097b0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:12:23.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.550+0000 7f19377fe700 1 -- 172.21.15.88:0/475126070 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f19500c7470 con 0x7f1950153010 2023-10-18T16:12:23.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.753+0000 7f19561bf700 1 -- 172.21.15.88:0/475126070 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f19500c7470 con 0x7f1950153010 2023-10-18T16:12:23.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.753+0000 7f19377fe700 1 -- 172.21.15.88:0/475126070 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7f19500c7470 con 0x7f1950153010 2023-10-18T16:12:23.757 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:12:23.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.755+0000 7f19561bf700 1 -- 172.21.15.88:0/475126070 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f193c05f150 msgr2=0x7f193c061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:23.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.755+0000 7f19561bf700 1 --2- 172.21.15.88:0/475126070 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f193c05f150 0x7f193c061610 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f19400075a0 tx=0x7f19400097b0).stop 2023-10-18T16:12:23.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.755+0000 7f19561bf700 1 -- 172.21.15.88:0/475126070 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1950153010 msgr2=0x7f1950163900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:23.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.755+0000 7f19561bf700 1 --2- 172.21.15.88:0/475126070 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1950153010 0x7f1950163900 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7f1950084410 tx=0x7f1948017000).stop 2023-10-18T16:12:23.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.755+0000 7f19561bf700 1 -- 172.21.15.88:0/475126070 shutdown_connections 2023-10-18T16:12:23.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.755+0000 7f19561bf700 1 --2- 172.21.15.88:0/475126070 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f193c05f150 0x7f193c061610 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:23.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.755+0000 7f19561bf700 1 --2- 172.21.15.88:0/475126070 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1950153010 0x7f1950163900 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:23.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.755+0000 7f19561bf700 1 --2- 172.21.15.88:0/475126070 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1950064d70 0x7f195015e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:23.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.755+0000 7f19561bf700 1 --2- 172.21.15.88:0/475126070 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f19501525f0 0x7f195015e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:23.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.755+0000 7f19561bf700 1 -- 172.21.15.88:0/475126070 >> 172.21.15.88:0/475126070 conn(0x7f1950069d80 msgr2=0x7f1950155af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:12:23.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.755+0000 7f19561bf700 1 -- 172.21.15.88:0/475126070 shutdown_connections 2023-10-18T16:12:23.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:23.755+0000 7f19561bf700 1 -- 172.21.15.88:0/475126070 wait complete. 2023-10-18T16:12:23.770 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:12:26.289 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~170s 2023-10-18T16:12:26.290 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~183s 2023-10-18T16:12:26.290 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~216s 2023-10-18T16:12:26.290 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~196s 2023-10-18T16:12:26.290 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~277s 2023-10-18T16:12:26.290 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.f is failed for ~230s 2023-10-18T16:12:26.771 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:12:26.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.893+0000 7f813e798700 1 Processor -- start 2023-10-18T16:12:26.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.893+0000 7f813e798700 1 -- start start 2023-10-18T16:12:26.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.894+0000 7f813e798700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f813814d8c0 0x7f8138064440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:26.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.894+0000 7f813e798700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f81380a9050 0x7f8138064930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:26.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.894+0000 7f813e798700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8138064150 con 0x7f8138065360 2023-10-18T16:12:26.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.894+0000 7f813e798700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8138153a40 con 0x7f81380a9050 2023-10-18T16:12:26.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.894+0000 7f813e798700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8138063e70 con 0x7f813814d8c0 2023-10-18T16:12:26.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.894+0000 7f813cd35700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f8138065360 0x7f8138153ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:44940/0 (socket says 172.21.15.88:44940) 2023-10-18T16:12:26.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.894+0000 7f813cd35700 1 -- 172.21.15.88:0/970075158 learned_addr learned my addr 172.21.15.88:0/970075158 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:12:26.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.894+0000 7f8137fff700 1 --2- 172.21.15.88:0/970075158 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f813814d8c0 0x7f8138064440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:26.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.894+0000 7f81377fe700 1 --2- 172.21.15.88:0/970075158 >> v2:172.21.15.151:3300/0 conn(0x7f81380a9050 0x7f8138064930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:26.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.894+0000 7f8136ffd700 1 -- 172.21.15.88:0/970075158 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2400856563 0 0) 0x7f8138064150 con 0x7f8138065360 2023-10-18T16:12:26.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.894+0000 7f8136ffd700 1 -- 172.21.15.88:0/970075158 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81200033f0 con 0x7f8138065360 2023-10-18T16:12:26.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.894+0000 7f8136ffd700 1 -- 172.21.15.88:0/970075158 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2034078279 0 0) 0x7f81200033f0 con 0x7f8138065360 2023-10-18T16:12:26.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.894+0000 7f8136ffd700 1 -- 172.21.15.88:0/970075158 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f813814d8c0 msgr2=0x7f8138064440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:12:26.905 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.894+0000 7f8136ffd700 1 --2- 172.21.15.88:0/970075158 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f813814d8c0 0x7f8138064440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:26.905 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.894+0000 7f8136ffd700 1 -- 172.21.15.88:0/970075158 >> v2:172.21.15.151:3300/0 conn(0x7f81380a9050 msgr2=0x7f8138064930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:12:26.905 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.894+0000 7f8136ffd700 1 --2- 172.21.15.88:0/970075158 >> v2:172.21.15.151:3300/0 conn(0x7f81380a9050 0x7f8138064930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:26.905 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.894+0000 7f8136ffd700 1 -- 172.21.15.88:0/970075158 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81380c72c0 con 0x7f8138065360 2023-10-18T16:12:26.905 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.894+0000 7f8136ffd700 1 -- 172.21.15.88:0/970075158 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441715075 0 0) 0x7f81300017f0 con 0x7f8138065360 2023-10-18T16:12:26.906 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.894+0000 7f8136ffd700 1 -- 172.21.15.88:0/970075158 >> v1:172.21.15.88:6789/0 conn(0x7f8138065360 legacy=0x7f8138153ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:26.906 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.895+0000 7f8136ffd700 1 --2- 172.21.15.88:0/970075158 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f81200038e0 0x7f8120003d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:26.906 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.895+0000 7f8136ffd700 1 --2- 172.21.15.88:0/970075158 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f813814d8c0 0x7f8120004690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:26.906 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.895+0000 7f8136ffd700 1 --2- 172.21.15.88:0/970075158 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8120004b80 0x7f8120005000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:26.906 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.895+0000 7f8136ffd700 1 -- 172.21.15.88:0/970075158 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8138153a40 con 0x7f8120004b80 2023-10-18T16:12:26.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.895+0000 7f8136ffd700 1 -- 172.21.15.88:0/970075158 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8138063e70 con 0x7f81200038e0 2023-10-18T16:12:26.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.895+0000 7f8136ffd700 1 -- 172.21.15.88:0/970075158 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f812000b8e0 con 0x7f813814d8c0 2023-10-18T16:12:26.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.895+0000 7f813cd35700 1 --2- 172.21.15.88:0/970075158 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8120004b80 0x7f8120005000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:26.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.895+0000 7f8137fff700 1 --2- 172.21.15.88:0/970075158 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f81200038e0 0x7f8120003d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:26.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.895+0000 7f81377fe700 1 --2- 172.21.15.88:0/970075158 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f813814d8c0 0x7f8120004690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:26.908 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.895+0000 7f813cd35700 1 -- 172.21.15.88:0/970075158 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f813814d8c0 msgr2=0x7f8120004690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:12:26.908 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.895+0000 7f813cd35700 1 --2- 172.21.15.88:0/970075158 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f813814d8c0 0x7f8120004690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:26.908 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.895+0000 7f813cd35700 1 -- 172.21.15.88:0/970075158 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f81200038e0 msgr2=0x7f8120003d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:12:26.908 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.895+0000 7f813cd35700 1 --2- 172.21.15.88:0/970075158 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f81200038e0 0x7f8120003d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:26.908 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.895+0000 7f813cd35700 1 -- 172.21.15.88:0/970075158 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81200033f0 con 0x7f8120004b80 2023-10-18T16:12:26.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.895+0000 7f813cd35700 1 --2- 172.21.15.88:0/970075158 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8120004b80 0x7f8120005000 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7f813000c530 tx=0x7f8130002210).ready entity=mon.0 client_cookie=885359ed8f1ede88 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:12:26.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.895+0000 7f8136ffd700 1 -- 172.21.15.88:0/970075158 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8130007de0 con 0x7f8120004b80 2023-10-18T16:12:26.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.895+0000 7f8136ffd700 1 -- 172.21.15.88:0/970075158 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f813000ce60 con 0x7f8120004b80 2023-10-18T16:12:26.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.895+0000 7f8136ffd700 1 -- 172.21.15.88:0/970075158 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8130005d50 con 0x7f8120004b80 2023-10-18T16:12:26.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.896+0000 7f813e798700 1 -- 172.21.15.88:0/970075158 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8120004b80 msgr2=0x7f8120005000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:26.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.896+0000 7f813e798700 1 --2- 172.21.15.88:0/970075158 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8120004b80 0x7f8120005000 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7f813000c530 tx=0x7f8130002210).stop 2023-10-18T16:12:26.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.896+0000 7f813cd35700 1 -- 172.21.15.88:0/970075158 reap_dead start 2023-10-18T16:12:26.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.896+0000 7f813e798700 1 -- 172.21.15.88:0/970075158 shutdown_connections 2023-10-18T16:12:26.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.896+0000 7f813e798700 1 -- 172.21.15.88:0/970075158 >> 172.21.15.88:0/970075158 conn(0x7f813806a810 msgr2=0x7f81380c6a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:12:26.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.896+0000 7f813e798700 1 -- 172.21.15.88:0/970075158 shutdown_connections 2023-10-18T16:12:26.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.896+0000 7f813e798700 1 -- 172.21.15.88:0/970075158 wait complete. 2023-10-18T16:12:26.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.897+0000 7f813e798700 1 Processor -- start 2023-10-18T16:12:26.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.897+0000 7f813e798700 1 -- start start 2023-10-18T16:12:26.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.898+0000 7f813e798700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8120004b80 0x7f8138160450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:26.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.898+0000 7f813e798700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f813814eac0 0x7f813814af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:26.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.898+0000 7f813e798700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f81380aa210 0x7f81380aa670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:26.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.898+0000 7f813e798700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8138155d10 con 0x7f81380aa210 2023-10-18T16:12:26.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.898+0000 7f813e798700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8138150520 con 0x7f813814eac0 2023-10-18T16:12:26.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.898+0000 7f813e798700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8138150800 con 0x7f8120004b80 2023-10-18T16:12:26.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.898+0000 7f813cd35700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f81380aa210 0x7f81380aa670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:26.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.898+0000 7f813cd35700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f81380aa210 0x7f81380aa670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35028/0 (socket says 172.21.15.88:35028) 2023-10-18T16:12:26.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.898+0000 7f813cd35700 1 -- 172.21.15.88:0/1723950753 learned_addr learned my addr 172.21.15.88:0/1723950753 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:12:26.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.898+0000 7f81377fe700 1 --2- 172.21.15.88:0/1723950753 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f813814eac0 0x7f813814af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:26.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.898+0000 7f8137fff700 1 --2- 172.21.15.88:0/1723950753 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8120004b80 0x7f8138160450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:26.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.898+0000 7f813cd35700 1 -- 172.21.15.88:0/1723950753 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8120004b80 msgr2=0x7f8138160450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:26.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.898+0000 7f813cd35700 1 --2- 172.21.15.88:0/1723950753 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8120004b80 0x7f8138160450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:26.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.898+0000 7f813cd35700 1 -- 172.21.15.88:0/1723950753 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f813814eac0 msgr2=0x7f813814af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:26.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.898+0000 7f813cd35700 1 --2- 172.21.15.88:0/1723950753 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f813814eac0 0x7f813814af80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:26.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.898+0000 7f813cd35700 1 -- 172.21.15.88:0/1723950753 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8138150ae0 con 0x7f81380aa210 2023-10-18T16:12:26.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.898+0000 7f813cd35700 1 --2- 172.21.15.88:0/1723950753 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f81380aa210 0x7f81380aa670 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f813000d010 tx=0x7f8130015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:12:26.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.898+0000 7f811ffff700 1 -- 172.21.15.88:0/1723950753 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8130007de0 con 0x7f81380aa210 2023-10-18T16:12:26.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.898+0000 7f811ffff700 1 -- 172.21.15.88:0/1723950753 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81300074f0 con 0x7f81380aa210 2023-10-18T16:12:26.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.898+0000 7f813e798700 1 -- 172.21.15.88:0/1723950753 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8138150dc0 con 0x7f81380aa210 2023-10-18T16:12:26.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.898+0000 7f811ffff700 1 -- 172.21.15.88:0/1723950753 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81300078d0 con 0x7f81380aa210 2023-10-18T16:12:26.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.898+0000 7f813e798700 1 -- 172.21.15.88:0/1723950753 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f813814e2c0 con 0x7f81380aa210 2023-10-18T16:12:26.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.899+0000 7f813e798700 1 -- 172.21.15.88:0/1723950753 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f813814e910 con 0x7f81380aa210 2023-10-18T16:12:26.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.900+0000 7f811ffff700 1 -- 172.21.15.88:0/1723950753 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f8130020410 con 0x7f81380aa210 2023-10-18T16:12:26.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.900+0000 7f811ffff700 1 --2- 172.21.15.88:0/1723950753 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f812405f180 0x7f8124061640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:26.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.900+0000 7f811ffff700 1 -- 172.21.15.88:0/1723950753 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6725+0+0 (secure 0 0 0) 0x7f8130019030 con 0x7f81380aa210 2023-10-18T16:12:26.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.900+0000 7f8137fff700 1 --2- 172.21.15.88:0/1723950753 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f812405f180 0x7f8124061640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:26.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.901+0000 7f8137fff700 1 --2- 172.21.15.88:0/1723950753 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f812405f180 0x7f8124061640 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f8128006d80 tx=0x7f8128009060).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:12:26.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:26.904+0000 7f811ffff700 1 -- 172.21.15.88:0/1723950753 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f813004c330 con 0x7f81380aa210 2023-10-18T16:12:27.107 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:27.103+0000 7f813e798700 1 -- 172.21.15.88:0/1723950753 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f81380c6500 con 0x7f81380aa210 2023-10-18T16:12:27.107 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:27.104+0000 7f811ffff700 1 -- 172.21.15.88:0/1723950753 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7f813004f950 con 0x7f81380aa210 2023-10-18T16:12:27.108 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:12:27.109 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:27.105+0000 7f813e798700 1 -- 172.21.15.88:0/1723950753 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f812405f180 msgr2=0x7f8124061640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:27.109 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:27.105+0000 7f813e798700 1 --2- 172.21.15.88:0/1723950753 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f812405f180 0x7f8124061640 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f8128006d80 tx=0x7f8128009060).stop 2023-10-18T16:12:27.109 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:27.105+0000 7f813e798700 1 -- 172.21.15.88:0/1723950753 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f81380aa210 msgr2=0x7f81380aa670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:27.110 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:27.105+0000 7f813e798700 1 --2- 172.21.15.88:0/1723950753 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f81380aa210 0x7f81380aa670 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f813000d010 tx=0x7f8130015000).stop 2023-10-18T16:12:27.110 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:27.105+0000 7f813e798700 1 -- 172.21.15.88:0/1723950753 shutdown_connections 2023-10-18T16:12:27.110 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:27.105+0000 7f813e798700 1 --2- 172.21.15.88:0/1723950753 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f812405f180 0x7f8124061640 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:27.110 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:27.105+0000 7f813e798700 1 --2- 172.21.15.88:0/1723950753 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f81380aa210 0x7f81380aa670 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:27.110 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:27.105+0000 7f813e798700 1 --2- 172.21.15.88:0/1723950753 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8120004b80 0x7f8138160450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:27.111 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:27.105+0000 7f813e798700 1 --2- 172.21.15.88:0/1723950753 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f813814eac0 0x7f813814af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:27.111 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:27.105+0000 7f813e798700 1 -- 172.21.15.88:0/1723950753 >> 172.21.15.88:0/1723950753 conn(0x7f813806a810 msgr2=0x7f81380c5df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:12:27.111 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:27.105+0000 7f813e798700 1 -- 172.21.15.88:0/1723950753 shutdown_connections 2023-10-18T16:12:27.111 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:27.105+0000 7f813e798700 1 -- 172.21.15.88:0/1723950753 wait complete. 2023-10-18T16:12:27.120 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:12:30.121 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:12:30.243 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.239+0000 7f70a94a5700 1 Processor -- start 2023-10-18T16:12:30.243 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.239+0000 7f70a94a5700 1 -- start start 2023-10-18T16:12:30.246 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.240+0000 7f70a94a5700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f70a4063d70 0x7f70a40c6e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:30.246 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.240+0000 7f70a94a5700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f70a40c2400 0x7f70a40c2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:30.246 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.240+0000 7f70a94a5700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70a4151020 con 0x7f70a40c6a60 2023-10-18T16:12:30.246 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.240+0000 7f70a94a5700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70a4150090 con 0x7f70a4063d70 2023-10-18T16:12:30.246 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.240+0000 7f70a94a5700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70a40c2d80 con 0x7f70a40c2400 2023-10-18T16:12:30.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.240+0000 7f70a37fe700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f70a4063d70 0x7f70a40c6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:30.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.240+0000 7f70a37fe700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f70a4063d70 0x7f70a40c6e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:51700/0 (socket says 172.21.15.88:51700) 2023-10-18T16:12:30.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.240+0000 7f70a37fe700 1 -- 172.21.15.88:0/2401178939 learned_addr learned my addr 172.21.15.88:0/2401178939 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:12:30.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.240+0000 7f70a37fe700 1 -- 172.21.15.88:0/2401178939 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f70a40c2400 msgr2=0x7f70a40c2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:12:30.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.240+0000 7f70a8ca4700 1 --2- 172.21.15.88:0/2401178939 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f70a40c2400 0x7f70a40c2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:30.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.240+0000 7f70a37fe700 1 --2- 172.21.15.88:0/2401178939 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f70a40c2400 0x7f70a40c2890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:30.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.240+0000 7f70a37fe700 1 -- 172.21.15.88:0/2401178939 >> v1:172.21.15.151:6789/0 conn(0x7f70a40c6a60 legacy=0x7f70a4153660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:30.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.240+0000 7f70a37fe700 1 -- 172.21.15.88:0/2401178939 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70a4150b90 con 0x7f70a4063d70 2023-10-18T16:12:30.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.240+0000 7f70a8ca4700 1 --2- 172.21.15.88:0/2401178939 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f70a40c2400 0x7f70a40c2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-10-18T16:12:30.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.241+0000 7f70a37fe700 1 --2- 172.21.15.88:0/2401178939 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70a4063d70 0x7f70a40c6e40 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f7098009830 tx=0x7f70980090f0).ready entity=mon.0 client_cookie=98e2942ea3554af4 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:12:30.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.241+0000 7f70a2ffd700 1 -- 172.21.15.88:0/2401178939 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f709800b600 con 0x7f70a4063d70 2023-10-18T16:12:30.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.241+0000 7f70a2ffd700 1 -- 172.21.15.88:0/2401178939 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f709800b760 con 0x7f70a4063d70 2023-10-18T16:12:30.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.241+0000 7f70a2ffd700 1 -- 172.21.15.88:0/2401178939 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f709800bae0 con 0x7f70a4063d70 2023-10-18T16:12:30.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.241+0000 7f70a94a5700 1 -- 172.21.15.88:0/2401178939 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70a4063d70 msgr2=0x7f70a40c6e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:30.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.241+0000 7f70a94a5700 1 --2- 172.21.15.88:0/2401178939 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70a4063d70 0x7f70a40c6e40 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f7098009830 tx=0x7f70980090f0).stop 2023-10-18T16:12:30.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.241+0000 7f70a94a5700 1 -- 172.21.15.88:0/2401178939 shutdown_connections 2023-10-18T16:12:30.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.241+0000 7f70a94a5700 1 --2- 172.21.15.88:0/2401178939 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f70a40c2400 0x7f70a40c2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:30.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.241+0000 7f70a94a5700 1 --2- 172.21.15.88:0/2401178939 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70a4063d70 0x7f70a40c6e40 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:30.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.241+0000 7f70a94a5700 1 -- 172.21.15.88:0/2401178939 >> 172.21.15.88:0/2401178939 conn(0x7f70a405e050 msgr2=0x7f70a405e460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:12:30.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.241+0000 7f70a94a5700 1 -- 172.21.15.88:0/2401178939 shutdown_connections 2023-10-18T16:12:30.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.241+0000 7f70a94a5700 1 -- 172.21.15.88:0/2401178939 wait complete. 2023-10-18T16:12:30.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.242+0000 7f70a94a5700 1 Processor -- start 2023-10-18T16:12:30.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.243+0000 7f70a94a5700 1 -- start start 2023-10-18T16:12:30.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.243+0000 7f70a94a5700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70a4063d70 0x7f70a4160940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:30.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.243+0000 7f70a94a5700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f70a40c2400 0x7f70a41521c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:30.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.243+0000 7f70a94a5700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f70a40c6a60 0x7f70a40c1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:30.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.243+0000 7f70a94a5700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70a4151490 con 0x7f70a4063d70 2023-10-18T16:12:30.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.243+0000 7f70a94a5700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70a41526b0 con 0x7f70a40c6a60 2023-10-18T16:12:30.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.243+0000 7f70a94a5700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70a414eaf0 con 0x7f70a40c2400 2023-10-18T16:12:30.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.243+0000 7f70a3fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70a4063d70 0x7f70a4160940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:30.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.243+0000 7f70a8ca4700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f70a40c6a60 0x7f70a40c1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:30.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.243+0000 7f70a3fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70a4063d70 0x7f70a4160940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:51704/0 (socket says 172.21.15.88:51704) 2023-10-18T16:12:30.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.243+0000 7f70a3fff700 1 -- 172.21.15.88:0/681592207 learned_addr learned my addr 172.21.15.88:0/681592207 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:12:30.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.243+0000 7f70a37fe700 1 --2- 172.21.15.88:0/681592207 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f70a40c2400 0x7f70a41521c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:30.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.244+0000 7f70a3fff700 1 -- 172.21.15.88:0/681592207 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f70a40c2400 msgr2=0x7f70a41521c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:30.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.244+0000 7f70a3fff700 1 --2- 172.21.15.88:0/681592207 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f70a40c2400 0x7f70a41521c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:30.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.244+0000 7f70a3fff700 1 -- 172.21.15.88:0/681592207 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f70a40c6a60 msgr2=0x7f70a40c1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:30.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.244+0000 7f70a3fff700 1 --2- 172.21.15.88:0/681592207 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f70a40c6a60 0x7f70a40c1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:30.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.244+0000 7f70a3fff700 1 -- 172.21.15.88:0/681592207 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7098009370 con 0x7f70a4063d70 2023-10-18T16:12:30.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.244+0000 7f70a3fff700 1 --2- 172.21.15.88:0/681592207 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70a4063d70 0x7f70a4160940 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f709400c440 tx=0x7f7094007db0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:12:30.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.244+0000 7f708bfff700 1 -- 172.21.15.88:0/681592207 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70940123f0 con 0x7f70a4063d70 2023-10-18T16:12:30.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.244+0000 7f708bfff700 1 -- 172.21.15.88:0/681592207 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7094001e90 con 0x7f70a4063d70 2023-10-18T16:12:30.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.244+0000 7f70a94a5700 1 -- 172.21.15.88:0/681592207 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70a40c38b0 con 0x7f70a4063d70 2023-10-18T16:12:30.258 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.244+0000 7f708bfff700 1 -- 172.21.15.88:0/681592207 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7094012780 con 0x7f70a4063d70 2023-10-18T16:12:30.258 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.244+0000 7f70a94a5700 1 -- 172.21.15.88:0/681592207 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70a41516e0 con 0x7f70a4063d70 2023-10-18T16:12:30.258 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.245+0000 7f70a94a5700 1 -- 172.21.15.88:0/681592207 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70a4084340 con 0x7f70a4063d70 2023-10-18T16:12:30.258 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.245+0000 7f708bfff700 1 -- 172.21.15.88:0/681592207 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f7094012b30 con 0x7f70a4063d70 2023-10-18T16:12:30.258 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.246+0000 7f708bfff700 1 --2- 172.21.15.88:0/681592207 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f709005f120 0x7f70900615e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:30.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.246+0000 7f708bfff700 1 -- 172.21.15.88:0/681592207 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6725+0+0 (secure 0 0 0) 0x7f7094079550 con 0x7f70a4063d70 2023-10-18T16:12:30.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.246+0000 7f70a37fe700 1 --2- 172.21.15.88:0/681592207 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f709005f120 0x7f70900615e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:30.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.246+0000 7f70a37fe700 1 --2- 172.21.15.88:0/681592207 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f709005f120 0x7f70900615e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f7098009130 tx=0x7f7098017000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:12:30.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.250+0000 7f708bfff700 1 -- 172.21.15.88:0/681592207 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f7094047ac0 con 0x7f70a4063d70 2023-10-18T16:12:30.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.451+0000 7f70a94a5700 1 -- 172.21.15.88:0/681592207 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f70a4151020 con 0x7f70a4063d70 2023-10-18T16:12:30.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.451+0000 7f708bfff700 1 -- 172.21.15.88:0/681592207 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7f70a4151020 con 0x7f70a4063d70 2023-10-18T16:12:30.455 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:12:30.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.453+0000 7f70a94a5700 1 -- 172.21.15.88:0/681592207 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f709005f120 msgr2=0x7f70900615e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:30.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.453+0000 7f70a94a5700 1 --2- 172.21.15.88:0/681592207 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f709005f120 0x7f70900615e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f7098009130 tx=0x7f7098017000).stop 2023-10-18T16:12:30.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.453+0000 7f70a94a5700 1 -- 172.21.15.88:0/681592207 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70a4063d70 msgr2=0x7f70a4160940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:30.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.453+0000 7f70a94a5700 1 --2- 172.21.15.88:0/681592207 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70a4063d70 0x7f70a4160940 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f709400c440 tx=0x7f7094007db0).stop 2023-10-18T16:12:30.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.453+0000 7f70a94a5700 1 -- 172.21.15.88:0/681592207 shutdown_connections 2023-10-18T16:12:30.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.453+0000 7f70a94a5700 1 --2- 172.21.15.88:0/681592207 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f709005f120 0x7f70900615e0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:30.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.453+0000 7f70a94a5700 1 --2- 172.21.15.88:0/681592207 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f70a40c6a60 0x7f70a40c1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:30.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.453+0000 7f70a94a5700 1 --2- 172.21.15.88:0/681592207 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70a4063d70 0x7f70a4160940 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:30.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.453+0000 7f70a94a5700 1 --2- 172.21.15.88:0/681592207 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f70a40c2400 0x7f70a41521c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:30.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.453+0000 7f70a94a5700 1 -- 172.21.15.88:0/681592207 >> 172.21.15.88:0/681592207 conn(0x7f70a405e050 msgr2=0x7f70a414c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:12:30.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.453+0000 7f70a94a5700 1 -- 172.21.15.88:0/681592207 shutdown_connections 2023-10-18T16:12:30.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:30.453+0000 7f70a94a5700 1 -- 172.21.15.88:0/681592207 wait complete. 2023-10-18T16:12:30.468 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:12:32.601 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~177s 2023-10-18T16:12:32.601 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~190s 2023-10-18T16:12:32.602 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~222s 2023-10-18T16:12:32.602 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~203s 2023-10-18T16:12:32.602 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~284s 2023-10-18T16:12:32.603 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.f is failed for ~236s 2023-10-18T16:12:33.470 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:12:33.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.591+0000 7f4ce45e2700 1 Processor -- start 2023-10-18T16:12:33.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.591+0000 7f4ce45e2700 1 -- start start 2023-10-18T16:12:33.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.592+0000 7f4ce45e2700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4cdc064ea0 0x7f4cdc151e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:33.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.592+0000 7f4ce45e2700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f4cdc1522f0 0x7f4cdc150640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:33.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.592+0000 7f4ce45e2700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cdc065330 con 0x7f4cdc0a6e80 2023-10-18T16:12:33.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.592+0000 7f4ce45e2700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cdc0a87f0 con 0x7f4cdc064ea0 2023-10-18T16:12:33.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.592+0000 7f4ce45e2700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cdc0a8ad0 con 0x7f4cdc1522f0 2023-10-18T16:12:33.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.592+0000 7f4ce1b7d700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4cdc064ea0 0x7f4cdc151e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:33.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.592+0000 7f4ce1b7d700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4cdc064ea0 0x7f4cdc151e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:51710/0 (socket says 172.21.15.88:51710) 2023-10-18T16:12:33.599 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.592+0000 7f4ce1b7d700 1 -- 172.21.15.88:0/2538667894 learned_addr learned my addr 172.21.15.88:0/2538667894 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:12:33.599 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.592+0000 7f4ce237e700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f4cdc0a6e80 0x7f4cdc14d8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:59090/0 (socket says 172.21.15.88:59090) 2023-10-18T16:12:33.599 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.592+0000 7f4ce2b7f700 1 --2- 172.21.15.88:0/2538667894 >> v2:172.21.15.151:3300/0 conn(0x7f4cdc1522f0 0x7f4cdc150640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:33.599 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.592+0000 7f4ce1b7d700 1 -- 172.21.15.88:0/2538667894 >> v2:172.21.15.151:3300/0 conn(0x7f4cdc1522f0 msgr2=0x7f4cdc150640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:12:33.599 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.592+0000 7f4ce1b7d700 1 --2- 172.21.15.88:0/2538667894 >> v2:172.21.15.151:3300/0 conn(0x7f4cdc1522f0 0x7f4cdc150640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:33.600 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.592+0000 7f4ce1b7d700 1 -- 172.21.15.88:0/2538667894 >> v1:172.21.15.88:6789/0 conn(0x7f4cdc0a6e80 legacy=0x7f4cdc14d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:33.600 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.592+0000 7f4ce137c700 1 -- 172.21.15.88:0/2538667894 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2530518954 0 0) 0x7f4cdc065330 con 0x7f4cdc0a6e80 2023-10-18T16:12:33.600 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.592+0000 7f4ce1b7d700 1 -- 172.21.15.88:0/2538667894 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cdc0a8db0 con 0x7f4cdc064ea0 2023-10-18T16:12:33.600 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.593+0000 7f4ce1b7d700 1 --2- 172.21.15.88:0/2538667894 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cdc064ea0 0x7f4cdc151e00 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f4cdc0c1040 tx=0x7f4ccc0090f0).ready entity=mon.0 client_cookie=26d684e680050944 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:12:33.600 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.593+0000 7f4ce137c700 1 -- 172.21.15.88:0/2538667894 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ccc00b520 con 0x7f4cdc064ea0 2023-10-18T16:12:33.601 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.593+0000 7f4ce137c700 1 -- 172.21.15.88:0/2538667894 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ccc00b680 con 0x7f4cdc064ea0 2023-10-18T16:12:33.601 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.593+0000 7f4ce137c700 1 -- 172.21.15.88:0/2538667894 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ccc00ba20 con 0x7f4cdc064ea0 2023-10-18T16:12:33.601 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.593+0000 7f4ce45e2700 1 -- 172.21.15.88:0/2538667894 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cdc064ea0 msgr2=0x7f4cdc151e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:33.601 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.593+0000 7f4ce45e2700 1 --2- 172.21.15.88:0/2538667894 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cdc064ea0 0x7f4cdc151e00 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f4cdc0c1040 tx=0x7f4ccc0090f0).stop 2023-10-18T16:12:33.601 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.593+0000 7f4ce45e2700 1 -- 172.21.15.88:0/2538667894 shutdown_connections 2023-10-18T16:12:33.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.593+0000 7f4ce45e2700 1 --2- 172.21.15.88:0/2538667894 >> v2:172.21.15.151:3300/0 conn(0x7f4cdc1522f0 0x7f4cdc150640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:33.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.593+0000 7f4ce45e2700 1 --2- 172.21.15.88:0/2538667894 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cdc064ea0 0x7f4cdc151e00 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:33.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.593+0000 7f4ce45e2700 1 -- 172.21.15.88:0/2538667894 >> 172.21.15.88:0/2538667894 conn(0x7f4cdc069d80 msgr2=0x7f4cdc06a190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:12:33.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.593+0000 7f4ce45e2700 1 -- 172.21.15.88:0/2538667894 shutdown_connections 2023-10-18T16:12:33.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.593+0000 7f4ce45e2700 1 -- 172.21.15.88:0/2538667894 wait complete. 2023-10-18T16:12:33.605 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.594+0000 7f4ce45e2700 1 Processor -- start 2023-10-18T16:12:33.605 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.594+0000 7f4ce45e2700 1 -- start start 2023-10-18T16:12:33.606 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.595+0000 7f4ce45e2700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4cdc064ea0 0x7f4cdc163e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:33.606 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.595+0000 7f4ce45e2700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4cdc0a6e80 0x7f4cdc1488a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:33.606 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.595+0000 7f4ce45e2700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cdc1522f0 0x7f4cdc06a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:33.606 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.595+0000 7f4ce45e2700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cdc0c3a40 con 0x7f4cdc1522f0 2023-10-18T16:12:33.606 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.595+0000 7f4ce45e2700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cdc0c2d40 con 0x7f4cdc064ea0 2023-10-18T16:12:33.607 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.595+0000 7f4ce45e2700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cdc1615c0 con 0x7f4cdc0a6e80 2023-10-18T16:12:33.607 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.595+0000 7f4ce2b7f700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cdc1522f0 0x7f4cdc06a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:33.607 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.595+0000 7f4ce2b7f700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cdc1522f0 0x7f4cdc06a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:51720/0 (socket says 172.21.15.88:51720) 2023-10-18T16:12:33.607 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.595+0000 7f4ce2b7f700 1 -- 172.21.15.88:0/1509292752 learned_addr learned my addr 172.21.15.88:0/1509292752 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:12:33.607 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.595+0000 7f4ce237e700 1 --2- 172.21.15.88:0/1509292752 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4cdc064ea0 0x7f4cdc163e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:33.607 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.595+0000 7f4ce1b7d700 1 --2- 172.21.15.88:0/1509292752 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4cdc0a6e80 0x7f4cdc1488a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:33.608 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.596+0000 7f4ce2b7f700 1 -- 172.21.15.88:0/1509292752 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4cdc0a6e80 msgr2=0x7f4cdc1488a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:33.608 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.596+0000 7f4ce2b7f700 1 --2- 172.21.15.88:0/1509292752 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4cdc0a6e80 0x7f4cdc1488a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:33.608 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.596+0000 7f4ce2b7f700 1 -- 172.21.15.88:0/1509292752 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4cdc064ea0 msgr2=0x7f4cdc163e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:33.608 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.596+0000 7f4ce2b7f700 1 --2- 172.21.15.88:0/1509292752 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4cdc064ea0 0x7f4cdc163e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:33.608 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.596+0000 7f4ce2b7f700 1 -- 172.21.15.88:0/1509292752 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ccc009370 con 0x7f4cdc1522f0 2023-10-18T16:12:33.609 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.596+0000 7f4ce2b7f700 1 --2- 172.21.15.88:0/1509292752 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cdc1522f0 0x7f4cdc06a160 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f4cd4009e20 tx=0x7f4cd400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:12:33.609 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.596+0000 7f4cd27fc700 1 -- 172.21.15.88:0/1509292752 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd400caa0 con 0x7f4cdc1522f0 2023-10-18T16:12:33.609 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.596+0000 7f4cd27fc700 1 -- 172.21.15.88:0/1509292752 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cd400f040 con 0x7f4cdc1522f0 2023-10-18T16:12:33.609 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.596+0000 7f4ce45e2700 1 -- 172.21.15.88:0/1509292752 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cdc164350 con 0x7f4cdc1522f0 2023-10-18T16:12:33.609 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.596+0000 7f4cd27fc700 1 -- 172.21.15.88:0/1509292752 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd400ce10 con 0x7f4cdc1522f0 2023-10-18T16:12:33.610 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.596+0000 7f4ce45e2700 1 -- 172.21.15.88:0/1509292752 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cdc0c2780 con 0x7f4cdc1522f0 2023-10-18T16:12:33.610 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.597+0000 7f4ce45e2700 1 -- 172.21.15.88:0/1509292752 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cdc06c4e0 con 0x7f4cdc1522f0 2023-10-18T16:12:33.610 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.597+0000 7f4cd27fc700 1 -- 172.21.15.88:0/1509292752 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f4cd4013610 con 0x7f4cdc1522f0 2023-10-18T16:12:33.610 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.597+0000 7f4cd27fc700 1 --2- 172.21.15.88:0/1509292752 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f4cc005f150 0x7f4cc0061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:33.610 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.598+0000 7f4cd27fc700 1 -- 172.21.15.88:0/1509292752 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6725+0+0 (secure 0 0 0) 0x7f4cd407aaf0 con 0x7f4cdc1522f0 2023-10-18T16:12:33.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.598+0000 7f4ce237e700 1 --2- 172.21.15.88:0/1509292752 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f4cc005f150 0x7f4cc0061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:33.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.598+0000 7f4ce237e700 1 --2- 172.21.15.88:0/1509292752 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f4cc005f150 0x7f4cc0061610 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f4cd80011e0 tx=0x7f4cd8007bd0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:12:33.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.602+0000 7f4cd27fc700 1 -- 172.21.15.88:0/1509292752 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f4cd404bb60 con 0x7f4cdc1522f0 2023-10-18T16:12:33.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.802+0000 7f4ce45e2700 1 -- 172.21.15.88:0/1509292752 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4cdc161700 con 0x7f4cdc1522f0 2023-10-18T16:12:33.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.803+0000 7f4cd27fc700 1 -- 172.21.15.88:0/1509292752 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7f4cd404f180 con 0x7f4cdc1522f0 2023-10-18T16:12:33.807 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:12:33.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.804+0000 7f4ce45e2700 1 -- 172.21.15.88:0/1509292752 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f4cc005f150 msgr2=0x7f4cc0061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:33.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.804+0000 7f4ce45e2700 1 --2- 172.21.15.88:0/1509292752 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f4cc005f150 0x7f4cc0061610 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f4cd80011e0 tx=0x7f4cd8007bd0).stop 2023-10-18T16:12:33.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.804+0000 7f4ce45e2700 1 -- 172.21.15.88:0/1509292752 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cdc1522f0 msgr2=0x7f4cdc06a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:33.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.804+0000 7f4ce45e2700 1 --2- 172.21.15.88:0/1509292752 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cdc1522f0 0x7f4cdc06a160 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f4cd4009e20 tx=0x7f4cd400b040).stop 2023-10-18T16:12:33.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.804+0000 7f4ce45e2700 1 -- 172.21.15.88:0/1509292752 shutdown_connections 2023-10-18T16:12:33.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.804+0000 7f4ce45e2700 1 --2- 172.21.15.88:0/1509292752 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f4cc005f150 0x7f4cc0061610 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:33.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.804+0000 7f4ce45e2700 1 --2- 172.21.15.88:0/1509292752 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cdc1522f0 0x7f4cdc06a160 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:33.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.804+0000 7f4ce45e2700 1 --2- 172.21.15.88:0/1509292752 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4cdc064ea0 0x7f4cdc163e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:33.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.804+0000 7f4ce45e2700 1 --2- 172.21.15.88:0/1509292752 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4cdc0a6e80 0x7f4cdc1488a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:33.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.804+0000 7f4ce45e2700 1 -- 172.21.15.88:0/1509292752 >> 172.21.15.88:0/1509292752 conn(0x7f4cdc069d80 msgr2=0x7f4cdc064790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:12:33.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.805+0000 7f4ce45e2700 1 -- 172.21.15.88:0/1509292752 shutdown_connections 2023-10-18T16:12:33.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:33.805+0000 7f4ce45e2700 1 -- 172.21.15.88:0/1509292752 wait complete. 2023-10-18T16:12:33.820 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:12:36.820 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:12:36.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:36.942+0000 7f5b15918700 1 Processor -- start 2023-10-18T16:12:36.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:36.943+0000 7f5b15918700 1 -- start start 2023-10-18T16:12:36.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:36.943+0000 7f5b15918700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5b10151910 0x7f5b10068650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:36.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:36.943+0000 7f5b15918700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f5b100c5660 0x7f5b10151cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:36.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:36.943+0000 7f5b15918700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b10067dd0 con 0x7f5b10068b40 2023-10-18T16:12:36.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:36.943+0000 7f5b15918700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b10067b20 con 0x7f5b10151910 2023-10-18T16:12:36.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:36.943+0000 7f5b15918700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b100b9b60 con 0x7f5b100c5660 2023-10-18T16:12:36.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:36.944+0000 7f5b0effd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5b10151910 0x7f5b10068650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:36.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:36.944+0000 7f5b0f7fe700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f5b10068b40 0x7f5b100a9720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:59106/0 (socket says 172.21.15.88:59106) 2023-10-18T16:12:36.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:36.944+0000 7f5b0effd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5b10151910 0x7f5b10068650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:51736/0 (socket says 172.21.15.88:51736) 2023-10-18T16:12:36.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:36.944+0000 7f5b0f7fe700 1 -- 172.21.15.88:0/519945178 learned_addr learned my addr 172.21.15.88:0/519945178 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:12:36.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:36.944+0000 7f5b067fc700 1 --2- 172.21.15.88:0/519945178 >> v2:172.21.15.151:3300/0 conn(0x7f5b100c5660 0x7f5b10151cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:36.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:36.944+0000 7f5b0effd700 1 -- 172.21.15.88:0/519945178 >> v2:172.21.15.151:3300/0 conn(0x7f5b100c5660 msgr2=0x7f5b10151cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:12:36.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:36.944+0000 7f5b0effd700 1 --2- 172.21.15.88:0/519945178 >> v2:172.21.15.151:3300/0 conn(0x7f5b100c5660 0x7f5b10151cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:36.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:36.944+0000 7f5b0effd700 1 -- 172.21.15.88:0/519945178 >> v1:172.21.15.88:6789/0 conn(0x7f5b10068b40 legacy=0x7f5b100a9720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:36.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:36.944+0000 7f5b0effd700 1 -- 172.21.15.88:0/519945178 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b100b9ff0 con 0x7f5b10151910 2023-10-18T16:12:36.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:36.944+0000 7f5b0e7fc700 1 -- 172.21.15.88:0/519945178 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3067665192 0 0) 0x7f5b10067dd0 con 0x7f5b10068b40 2023-10-18T16:12:36.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:36.944+0000 7f5b0effd700 1 --2- 172.21.15.88:0/519945178 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b10151910 0x7f5b10068650 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f5b100be590 tx=0x7f5b00009060).ready entity=mon.0 client_cookie=b009027dfee3f42e server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:12:36.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:36.944+0000 7f5b0e7fc700 1 -- 172.21.15.88:0/519945178 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b0000b610 con 0x7f5b10151910 2023-10-18T16:12:36.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:36.944+0000 7f5b0e7fc700 1 -- 172.21.15.88:0/519945178 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b0000b770 con 0x7f5b10151910 2023-10-18T16:12:36.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:36.944+0000 7f5b0e7fc700 1 -- 172.21.15.88:0/519945178 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b0000baf0 con 0x7f5b10151910 2023-10-18T16:12:36.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:36.945+0000 7f5b15918700 1 -- 172.21.15.88:0/519945178 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b10151910 msgr2=0x7f5b10068650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:36.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:36.945+0000 7f5b15918700 1 --2- 172.21.15.88:0/519945178 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b10151910 0x7f5b10068650 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f5b100be590 tx=0x7f5b00009060).stop 2023-10-18T16:12:36.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:36.945+0000 7f5b15918700 1 -- 172.21.15.88:0/519945178 shutdown_connections 2023-10-18T16:12:36.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:36.945+0000 7f5b15918700 1 --2- 172.21.15.88:0/519945178 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b10151910 0x7f5b10068650 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:36.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:36.945+0000 7f5b15918700 1 --2- 172.21.15.88:0/519945178 >> v2:172.21.15.151:3300/0 conn(0x7f5b100c5660 0x7f5b10151cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:36.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:36.945+0000 7f5b15918700 1 -- 172.21.15.88:0/519945178 >> 172.21.15.88:0/519945178 conn(0x7f5b100c4940 msgr2=0x7f5b100c4d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:12:36.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:36.945+0000 7f5b15918700 1 -- 172.21.15.88:0/519945178 shutdown_connections 2023-10-18T16:12:36.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:36.945+0000 7f5b15918700 1 -- 172.21.15.88:0/519945178 wait complete. 2023-10-18T16:12:36.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:36.946+0000 7f5b15918700 1 Processor -- start 2023-10-18T16:12:36.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:36.946+0000 7f5b15918700 1 -- start start 2023-10-18T16:12:36.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:36.947+0000 7f5b15918700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b10068b40 0x7f5b10146800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:36.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:36.947+0000 7f5b15918700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5b100c5660 0x7f5b1015fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:36.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:36.947+0000 7f5b15918700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5b10151910 0x7f5b1015ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:36.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:36.947+0000 7f5b15918700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1014de30 con 0x7f5b10068b40 2023-10-18T16:12:36.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:36.947+0000 7f5b15918700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1015f5f0 con 0x7f5b100c5660 2023-10-18T16:12:36.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:36.947+0000 7f5b15918700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b10160460 con 0x7f5b10151910 2023-10-18T16:12:36.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:36.947+0000 7f5b0effd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b10068b40 0x7f5b10146800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:36.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:36.947+0000 7f5b0effd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b10068b40 0x7f5b10146800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:51738/0 (socket says 172.21.15.88:51738) 2023-10-18T16:12:36.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:36.947+0000 7f5b0effd700 1 -- 172.21.15.88:0/4178994603 learned_addr learned my addr 172.21.15.88:0/4178994603 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:12:36.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:36.947+0000 7f5b067fc700 1 --2- 172.21.15.88:0/4178994603 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5b100c5660 0x7f5b1015fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:36.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:36.947+0000 7f5b0f7fe700 1 --2- 172.21.15.88:0/4178994603 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5b10151910 0x7f5b1015ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:36.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:36.947+0000 7f5b0effd700 1 -- 172.21.15.88:0/4178994603 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5b10151910 msgr2=0x7f5b1015ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:36.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:36.947+0000 7f5b0effd700 1 --2- 172.21.15.88:0/4178994603 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5b10151910 0x7f5b1015ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:36.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:36.947+0000 7f5b0effd700 1 -- 172.21.15.88:0/4178994603 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5b100c5660 msgr2=0x7f5b1015fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:36.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:36.947+0000 7f5b0effd700 1 --2- 172.21.15.88:0/4178994603 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5b100c5660 0x7f5b1015fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:36.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:36.947+0000 7f5b0effd700 1 -- 172.21.15.88:0/4178994603 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b000092a0 con 0x7f5b10068b40 2023-10-18T16:12:36.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:36.947+0000 7f5b0effd700 1 --2- 172.21.15.88:0/4178994603 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b10068b40 0x7f5b10146800 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f5b10069b10 tx=0x7f5b00017000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:12:36.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:36.947+0000 7f5b077fe700 1 -- 172.21.15.88:0/4178994603 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b0000b8e0 con 0x7f5b10068b40 2023-10-18T16:12:36.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:36.947+0000 7f5b077fe700 1 -- 172.21.15.88:0/4178994603 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b0000ba40 con 0x7f5b10068b40 2023-10-18T16:12:36.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:36.947+0000 7f5b15918700 1 -- 172.21.15.88:0/4178994603 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b10147520 con 0x7f5b10068b40 2023-10-18T16:12:36.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:36.947+0000 7f5b077fe700 1 -- 172.21.15.88:0/4178994603 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b0001b690 con 0x7f5b10068b40 2023-10-18T16:12:36.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:36.948+0000 7f5b15918700 1 -- 172.21.15.88:0/4178994603 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b1014c680 con 0x7f5b10068b40 2023-10-18T16:12:36.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:36.948+0000 7f5b15918700 1 -- 172.21.15.88:0/4178994603 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b1015f8d0 con 0x7f5b10068b40 2023-10-18T16:12:36.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:36.949+0000 7f5b077fe700 1 -- 172.21.15.88:0/4178994603 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f5b0001bac0 con 0x7f5b10068b40 2023-10-18T16:12:36.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:36.949+0000 7f5b077fe700 1 --2- 172.21.15.88:0/4178994603 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f5af005f150 0x7f5af0061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:36.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:36.949+0000 7f5b077fe700 1 -- 172.21.15.88:0/4178994603 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6725+0+0 (secure 0 0 0) 0x7f5b0007be70 con 0x7f5b10068b40 2023-10-18T16:12:36.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:36.949+0000 7f5b067fc700 1 --2- 172.21.15.88:0/4178994603 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f5af005f150 0x7f5af0061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:36.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:36.950+0000 7f5b067fc700 1 --2- 172.21.15.88:0/4178994603 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f5af005f150 0x7f5af0061610 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f5b10147270 tx=0x7f5af8008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:12:36.969 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:36.953+0000 7f5b077fe700 1 -- 172.21.15.88:0/4178994603 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f5b0004cee0 con 0x7f5b10068b40 2023-10-18T16:12:37.161 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:37.157+0000 7f5b15918700 1 -- 172.21.15.88:0/4178994603 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f5b1006c4e0 con 0x7f5b10068b40 2023-10-18T16:12:37.161 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:37.158+0000 7f5b077fe700 1 -- 172.21.15.88:0/4178994603 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7f5b00006b30 con 0x7f5b10068b40 2023-10-18T16:12:37.161 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:12:37.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:37.159+0000 7f5b15918700 1 -- 172.21.15.88:0/4178994603 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f5af005f150 msgr2=0x7f5af0061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:37.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:37.159+0000 7f5b15918700 1 --2- 172.21.15.88:0/4178994603 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f5af005f150 0x7f5af0061610 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f5b10147270 tx=0x7f5af8008040).stop 2023-10-18T16:12:37.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:37.159+0000 7f5b15918700 1 -- 172.21.15.88:0/4178994603 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b10068b40 msgr2=0x7f5b10146800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:37.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:37.159+0000 7f5b15918700 1 --2- 172.21.15.88:0/4178994603 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b10068b40 0x7f5b10146800 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f5b10069b10 tx=0x7f5b00017000).stop 2023-10-18T16:12:37.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:37.159+0000 7f5b15918700 1 -- 172.21.15.88:0/4178994603 shutdown_connections 2023-10-18T16:12:37.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:37.159+0000 7f5b15918700 1 --2- 172.21.15.88:0/4178994603 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f5af005f150 0x7f5af0061610 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:37.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:37.159+0000 7f5b15918700 1 --2- 172.21.15.88:0/4178994603 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5b10151910 0x7f5b1015ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:37.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:37.159+0000 7f5b15918700 1 --2- 172.21.15.88:0/4178994603 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b10068b40 0x7f5b10146800 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:37.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:37.159+0000 7f5b15918700 1 --2- 172.21.15.88:0/4178994603 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5b100c5660 0x7f5b1015fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:37.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:37.159+0000 7f5b15918700 1 -- 172.21.15.88:0/4178994603 >> 172.21.15.88:0/4178994603 conn(0x7f5b100c4940 msgr2=0x7f5b100befc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:12:37.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:37.159+0000 7f5b15918700 1 -- 172.21.15.88:0/4178994603 shutdown_connections 2023-10-18T16:12:37.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:37.159+0000 7f5b15918700 1 -- 172.21.15.88:0/4178994603 wait complete. 2023-10-18T16:12:37.174 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:12:38.855 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-18T16:12:38.858 DEBUG:teuthology.orchestra.run.smithi151:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-18T16:12:38.913 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~183s 2023-10-18T16:12:38.913 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~196s 2023-10-18T16:12:38.914 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~229s 2023-10-18T16:12:38.914 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~209s 2023-10-18T16:12:38.914 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~290s 2023-10-18T16:12:38.914 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.f is failed for ~242s 2023-10-18T16:12:40.175 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:12:40.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.295+0000 7f16f98a0700 1 Processor -- start 2023-10-18T16:12:40.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.295+0000 7f16f98a0700 1 -- start start 2023-10-18T16:12:40.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.296+0000 7f16f98a0700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f16f414d8c0 0x7f16f4064440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:40.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.296+0000 7f16f98a0700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16f4064150 con 0x7f16f40a9050 2023-10-18T16:12:40.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.296+0000 7f16f98a0700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16f4153a40 con 0x7f16f4065360 2023-10-18T16:12:40.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.296+0000 7f16f98a0700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16f4063e70 con 0x7f16f414d8c0 2023-10-18T16:12:40.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.296+0000 7f16ea7fc700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f16f40a9050 0x7f16f4153ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:48702/0 (socket says 172.21.15.88:48702) 2023-10-18T16:12:40.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.296+0000 7f16ea7fc700 1 -- 172.21.15.88:0/3625497304 learned_addr learned my addr 172.21.15.88:0/3625497304 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:12:40.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.296+0000 7f16f2ffd700 1 --2- 172.21.15.88:0/3625497304 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f16f414d8c0 0x7f16f4064440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:40.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.296+0000 7f16f27fc700 1 -- 172.21.15.88:0/3625497304 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4063737494 0 0) 0x7f16f4064150 con 0x7f16f40a9050 2023-10-18T16:12:40.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.296+0000 7f16f27fc700 1 -- 172.21.15.88:0/3625497304 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16e00033f0 con 0x7f16f40a9050 2023-10-18T16:12:40.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.296+0000 7f16f27fc700 1 -- 172.21.15.88:0/3625497304 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1294873345 0 0) 0x7f16e00033f0 con 0x7f16f40a9050 2023-10-18T16:12:40.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.296+0000 7f16f27fc700 1 -- 172.21.15.88:0/3625497304 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f16f414d8c0 msgr2=0x7f16f4064440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:12:40.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.296+0000 7f16f27fc700 1 --2- 172.21.15.88:0/3625497304 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f16f414d8c0 0x7f16f4064440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:40.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.296+0000 7f16f27fc700 1 -- 172.21.15.88:0/3625497304 >> v1:172.21.15.151:6789/0 conn(0x7f16f4065360 legacy=0x7f16f415acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:40.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.296+0000 7f16f27fc700 1 -- 172.21.15.88:0/3625497304 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f40c72c0 con 0x7f16f40a9050 2023-10-18T16:12:40.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.296+0000 7f16f27fc700 1 -- 172.21.15.88:0/3625497304 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441715075 0 0) 0x7f16dc001470 con 0x7f16f40a9050 2023-10-18T16:12:40.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.297+0000 7f16f27fc700 1 -- 172.21.15.88:0/3625497304 >> v1:172.21.15.88:6789/0 conn(0x7f16f40a9050 legacy=0x7f16f4153ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:40.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.297+0000 7f16f27fc700 1 --2- 172.21.15.88:0/3625497304 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f16f414d8c0 0x7f16e0003910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:40.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.297+0000 7f16f27fc700 1 --2- 172.21.15.88:0/3625497304 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16e0003e00 0x7f16e0004690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:40.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.297+0000 7f16f27fc700 1 --2- 172.21.15.88:0/3625497304 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f16e0004b80 0x7f16e0005000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:40.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.297+0000 7f16f27fc700 1 -- 172.21.15.88:0/3625497304 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f4063e70 con 0x7f16e0003e00 2023-10-18T16:12:40.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.297+0000 7f16f27fc700 1 -- 172.21.15.88:0/3625497304 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16e000b990 con 0x7f16e0004b80 2023-10-18T16:12:40.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.297+0000 7f16f27fc700 1 -- 172.21.15.88:0/3625497304 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16e000bb50 con 0x7f16f414d8c0 2023-10-18T16:12:40.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.297+0000 7f16ea7fc700 1 --2- 172.21.15.88:0/3625497304 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16e0003e00 0x7f16e0004690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:40.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.297+0000 7f16f2ffd700 1 --2- 172.21.15.88:0/3625497304 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f16f414d8c0 0x7f16e0003910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:40.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.297+0000 7f16f37fe700 1 --2- 172.21.15.88:0/3625497304 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f16e0004b80 0x7f16e0005000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:40.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.297+0000 7f16ea7fc700 1 -- 172.21.15.88:0/3625497304 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f16f414d8c0 msgr2=0x7f16e0003910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:12:40.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.297+0000 7f16ea7fc700 1 --2- 172.21.15.88:0/3625497304 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f16f414d8c0 0x7f16e0003910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:40.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.297+0000 7f16ea7fc700 1 -- 172.21.15.88:0/3625497304 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f16e0004b80 msgr2=0x7f16e0005000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:12:40.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.297+0000 7f16ea7fc700 1 --2- 172.21.15.88:0/3625497304 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f16e0004b80 0x7f16e0005000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:40.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.297+0000 7f16ea7fc700 1 -- 172.21.15.88:0/3625497304 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16e00033f0 con 0x7f16e0003e00 2023-10-18T16:12:40.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.297+0000 7f16f37fe700 1 --2- 172.21.15.88:0/3625497304 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f16e0004b80 0x7f16e0005000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:12:40.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.297+0000 7f16ea7fc700 1 --2- 172.21.15.88:0/3625497304 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16e0003e00 0x7f16e0004690 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f16dc00c160 tx=0x7f16dc001e70).ready entity=mon.0 client_cookie=f74f596e17b6f39c server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:12:40.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.297+0000 7f16f27fc700 1 -- 172.21.15.88:0/3625497304 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16dc007a40 con 0x7f16e0003e00 2023-10-18T16:12:40.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.297+0000 7f16f27fc700 1 -- 172.21.15.88:0/3625497304 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16dc00ca90 con 0x7f16e0003e00 2023-10-18T16:12:40.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.297+0000 7f16f27fc700 1 -- 172.21.15.88:0/3625497304 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16dc00ce10 con 0x7f16e0003e00 2023-10-18T16:12:40.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.298+0000 7f16f98a0700 1 -- 172.21.15.88:0/3625497304 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16e0003e00 msgr2=0x7f16e0004690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:40.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.298+0000 7f16f98a0700 1 --2- 172.21.15.88:0/3625497304 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16e0003e00 0x7f16e0004690 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f16dc00c160 tx=0x7f16dc001e70).stop 2023-10-18T16:12:40.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.298+0000 7f16f37fe700 1 -- 172.21.15.88:0/3625497304 reap_dead start 2023-10-18T16:12:40.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.298+0000 7f16f98a0700 1 -- 172.21.15.88:0/3625497304 shutdown_connections 2023-10-18T16:12:40.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.298+0000 7f16f98a0700 1 -- 172.21.15.88:0/3625497304 >> 172.21.15.88:0/3625497304 conn(0x7f16f406a810 msgr2=0x7f16f40c6a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:12:40.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.298+0000 7f16f98a0700 1 -- 172.21.15.88:0/3625497304 shutdown_connections 2023-10-18T16:12:40.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.298+0000 7f16f98a0700 1 -- 172.21.15.88:0/3625497304 wait complete. 2023-10-18T16:12:40.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.299+0000 7f16f98a0700 1 Processor -- start 2023-10-18T16:12:40.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.299+0000 7f16f98a0700 1 -- start start 2023-10-18T16:12:40.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.300+0000 7f16f98a0700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16e0003e00 0x7f16f41634d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:40.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.300+0000 7f16f98a0700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f16f41639c0 0x7f16f40baee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:40.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.300+0000 7f16f98a0700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f16f40bca90 0x7f16f40bb3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:40.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.300+0000 7f16f98a0700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f4164130 con 0x7f16e0003e00 2023-10-18T16:12:40.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.300+0000 7f16f98a0700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f40aab10 con 0x7f16f41639c0 2023-10-18T16:12:40.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.300+0000 7f16f98a0700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16f40baa50 con 0x7f16f40bca90 2023-10-18T16:12:40.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.300+0000 7f16f2ffd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16e0003e00 0x7f16f41634d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:40.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.300+0000 7f16f2ffd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16e0003e00 0x7f16f41634d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:52798/0 (socket says 172.21.15.88:52798) 2023-10-18T16:12:40.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.300+0000 7f16f2ffd700 1 -- 172.21.15.88:0/2661437399 learned_addr learned my addr 172.21.15.88:0/2661437399 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:12:40.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.300+0000 7f16ea7fc700 1 --2- 172.21.15.88:0/2661437399 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f16f41639c0 0x7f16f40baee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:40.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.300+0000 7f16f37fe700 1 --2- 172.21.15.88:0/2661437399 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f16f40bca90 0x7f16f40bb3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:40.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.300+0000 7f16f2ffd700 1 -- 172.21.15.88:0/2661437399 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f16f40bca90 msgr2=0x7f16f40bb3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:40.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.300+0000 7f16f2ffd700 1 --2- 172.21.15.88:0/2661437399 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f16f40bca90 0x7f16f40bb3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:40.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.300+0000 7f16f2ffd700 1 -- 172.21.15.88:0/2661437399 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f16f41639c0 msgr2=0x7f16f40baee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:40.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.300+0000 7f16f2ffd700 1 --2- 172.21.15.88:0/2661437399 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f16f41639c0 0x7f16f40baee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:40.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.300+0000 7f16f2ffd700 1 -- 172.21.15.88:0/2661437399 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f40bb8c0 con 0x7f16e0003e00 2023-10-18T16:12:40.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.300+0000 7f16ea7fc700 1 --2- 172.21.15.88:0/2661437399 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f16f41639c0 0x7f16f40baee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:12:40.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.300+0000 7f16f2ffd700 1 --2- 172.21.15.88:0/2661437399 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16e0003e00 0x7f16f41634d0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f16e4011a20 tx=0x7f16e4009060).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:12:40.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.300+0000 7f16eb7fe700 1 -- 172.21.15.88:0/2661437399 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e400f940 con 0x7f16e0003e00 2023-10-18T16:12:40.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.300+0000 7f16eb7fe700 1 -- 172.21.15.88:0/2661437399 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16e400faa0 con 0x7f16e0003e00 2023-10-18T16:12:40.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.300+0000 7f16f98a0700 1 -- 172.21.15.88:0/2661437399 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16f4163e50 con 0x7f16e0003e00 2023-10-18T16:12:40.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.300+0000 7f16eb7fe700 1 -- 172.21.15.88:0/2661437399 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e400fe20 con 0x7f16e0003e00 2023-10-18T16:12:40.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.300+0000 7f16f98a0700 1 -- 172.21.15.88:0/2661437399 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16f40aa830 con 0x7f16e0003e00 2023-10-18T16:12:40.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.302+0000 7f16f98a0700 1 -- 172.21.15.88:0/2661437399 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16f40bad30 con 0x7f16e0003e00 2023-10-18T16:12:40.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.302+0000 7f16eb7fe700 1 -- 172.21.15.88:0/2661437399 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f16e4014650 con 0x7f16e0003e00 2023-10-18T16:12:40.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.302+0000 7f16eb7fe700 1 --2- 172.21.15.88:0/2661437399 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f16d405f150 0x7f16d4061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:40.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.302+0000 7f16eb7fe700 1 -- 172.21.15.88:0/2661437399 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6725+0+0 (secure 0 0 0) 0x7f16e407b1c0 con 0x7f16e0003e00 2023-10-18T16:12:40.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.303+0000 7f16ea7fc700 1 --2- 172.21.15.88:0/2661437399 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f16d405f150 0x7f16d4061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:40.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.303+0000 7f16ea7fc700 1 --2- 172.21.15.88:0/2661437399 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f16d405f150 0x7f16d4061610 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f16dc001eb0 tx=0x7f16dc015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:12:40.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.306+0000 7f16eb7fe700 1 -- 172.21.15.88:0/2661437399 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f16e404c110 con 0x7f16e0003e00 2023-10-18T16:12:40.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.507+0000 7f16f98a0700 1 -- 172.21.15.88:0/2661437399 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f16f406c4e0 con 0x7f16e0003e00 2023-10-18T16:12:40.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.507+0000 7f16eb7fe700 1 -- 172.21.15.88:0/2661437399 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7f16e4018020 con 0x7f16e0003e00 2023-10-18T16:12:40.511 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:12:40.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.509+0000 7f16f98a0700 1 -- 172.21.15.88:0/2661437399 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f16d405f150 msgr2=0x7f16d4061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:40.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.509+0000 7f16f98a0700 1 --2- 172.21.15.88:0/2661437399 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f16d405f150 0x7f16d4061610 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f16dc001eb0 tx=0x7f16dc015000).stop 2023-10-18T16:12:40.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.509+0000 7f16f98a0700 1 -- 172.21.15.88:0/2661437399 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16e0003e00 msgr2=0x7f16f41634d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:40.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.509+0000 7f16f98a0700 1 --2- 172.21.15.88:0/2661437399 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16e0003e00 0x7f16f41634d0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f16e4011a20 tx=0x7f16e4009060).stop 2023-10-18T16:12:40.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.509+0000 7f16f98a0700 1 -- 172.21.15.88:0/2661437399 shutdown_connections 2023-10-18T16:12:40.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.509+0000 7f16f98a0700 1 --2- 172.21.15.88:0/2661437399 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f16d405f150 0x7f16d4061610 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:40.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.509+0000 7f16f98a0700 1 --2- 172.21.15.88:0/2661437399 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f16f40bca90 0x7f16f40bb3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:40.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.509+0000 7f16f98a0700 1 --2- 172.21.15.88:0/2661437399 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16e0003e00 0x7f16f41634d0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:40.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.509+0000 7f16f98a0700 1 --2- 172.21.15.88:0/2661437399 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f16f41639c0 0x7f16f40baee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:40.515 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.509+0000 7f16f98a0700 1 -- 172.21.15.88:0/2661437399 >> 172.21.15.88:0/2661437399 conn(0x7f16f406a810 msgr2=0x7f16f4152660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:12:40.515 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.509+0000 7f16f98a0700 1 -- 172.21.15.88:0/2661437399 shutdown_connections 2023-10-18T16:12:40.515 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:40.509+0000 7f16f98a0700 1 -- 172.21.15.88:0/2661437399 wait complete. 2023-10-18T16:12:40.524 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:12:43.525 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:12:43.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.647+0000 7fe5a0bfa700 1 Processor -- start 2023-10-18T16:12:43.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.647+0000 7fe5a0bfa700 1 -- start start 2023-10-18T16:12:43.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.648+0000 7fe5a0bfa700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fe59c14d8c0 0x7fe59c064440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:43.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.648+0000 7fe5a0bfa700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe59c064150 con 0x7fe59c065360 2023-10-18T16:12:43.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.648+0000 7fe5a0bfa700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe59c153a40 con 0x7fe59c0a9050 2023-10-18T16:12:43.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.648+0000 7fe5a0bfa700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe59c063e70 con 0x7fe59c14d8c0 2023-10-18T16:12:43.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.648+0000 7fe59ad9d700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fe59c065360 0x7fe59c15acb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:48710/0 (socket says 172.21.15.88:48710) 2023-10-18T16:12:43.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.648+0000 7fe59ad9d700 1 -- 172.21.15.88:0/1155560845 learned_addr learned my addr 172.21.15.88:0/1155560845 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:12:43.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.648+0000 7fe59a59c700 1 --2- 172.21.15.88:0/1155560845 >> v2:172.21.15.151:3300/0 conn(0x7fe59c14d8c0 0x7fe59c064440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:43.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.648+0000 7fe59959a700 1 -- 172.21.15.88:0/1155560845 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3188877727 0 0) 0x7fe59c064150 con 0x7fe59c065360 2023-10-18T16:12:43.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.648+0000 7fe59959a700 1 -- 172.21.15.88:0/1155560845 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5880033f0 con 0x7fe59c065360 2023-10-18T16:12:43.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.648+0000 7fe59959a700 1 -- 172.21.15.88:0/1155560845 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2873514792 0 0) 0x7fe5880033f0 con 0x7fe59c065360 2023-10-18T16:12:43.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.648+0000 7fe59959a700 1 -- 172.21.15.88:0/1155560845 >> v2:172.21.15.151:3300/0 conn(0x7fe59c14d8c0 msgr2=0x7fe59c064440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:12:43.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.648+0000 7fe59959a700 1 --2- 172.21.15.88:0/1155560845 >> v2:172.21.15.151:3300/0 conn(0x7fe59c14d8c0 0x7fe59c064440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:43.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.648+0000 7fe59959a700 1 -- 172.21.15.88:0/1155560845 >> v1:172.21.15.151:6789/0 conn(0x7fe59c0a9050 legacy=0x7fe59c153ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:43.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.648+0000 7fe59959a700 1 -- 172.21.15.88:0/1155560845 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe59c0c72c0 con 0x7fe59c065360 2023-10-18T16:12:43.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.648+0000 7fe59959a700 1 -- 172.21.15.88:0/1155560845 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441715075 0 0) 0x7fe58c001470 con 0x7fe59c065360 2023-10-18T16:12:43.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.648+0000 7fe59959a700 1 -- 172.21.15.88:0/1155560845 >> v1:172.21.15.88:6789/0 conn(0x7fe59c065360 legacy=0x7fe59c15acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:43.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.648+0000 7fe59959a700 1 --2- 172.21.15.88:0/1155560845 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe5880038e0 0x7fe588003d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:43.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.648+0000 7fe59959a700 1 --2- 172.21.15.88:0/1155560845 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe588004690 0x7fe588004b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:43.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.649+0000 7fe59959a700 1 --2- 172.21.15.88:0/1155560845 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe588005050 0x7fe58800b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:43.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.649+0000 7fe59959a700 1 -- 172.21.15.88:0/1155560845 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe59c063e70 con 0x7fe588004690 2023-10-18T16:12:43.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.649+0000 7fe59959a700 1 -- 172.21.15.88:0/1155560845 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe58800bfe0 con 0x7fe588005050 2023-10-18T16:12:43.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.649+0000 7fe59959a700 1 -- 172.21.15.88:0/1155560845 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe58800c1a0 con 0x7fe5880038e0 2023-10-18T16:12:43.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.649+0000 7fe599d9b700 1 --2- 172.21.15.88:0/1155560845 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe588004690 0x7fe588004b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:43.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.649+0000 7fe59ad9d700 1 --2- 172.21.15.88:0/1155560845 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe588005050 0x7fe58800b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:43.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.649+0000 7fe59a59c700 1 --2- 172.21.15.88:0/1155560845 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe5880038e0 0x7fe588003d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:43.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.649+0000 7fe599d9b700 1 -- 172.21.15.88:0/1155560845 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe5880038e0 msgr2=0x7fe588003d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:12:43.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.649+0000 7fe599d9b700 1 --2- 172.21.15.88:0/1155560845 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe5880038e0 0x7fe588003d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:43.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.649+0000 7fe599d9b700 1 -- 172.21.15.88:0/1155560845 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe588005050 msgr2=0x7fe58800b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:12:43.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.649+0000 7fe599d9b700 1 --2- 172.21.15.88:0/1155560845 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe588005050 0x7fe58800b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:43.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.649+0000 7fe599d9b700 1 -- 172.21.15.88:0/1155560845 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5880033f0 con 0x7fe588004690 2023-10-18T16:12:43.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.649+0000 7fe59ad9d700 1 -- 172.21.15.88:0/1155560845 reap_dead start 2023-10-18T16:12:43.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.649+0000 7fe599d9b700 1 --2- 172.21.15.88:0/1155560845 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe588004690 0x7fe588004b10 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7fe584001700 tx=0x7fe584007bb0).ready entity=mon.0 client_cookie=5dccc89767d8e91b server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:12:43.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.649+0000 7fe59959a700 1 -- 172.21.15.88:0/1155560845 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe584007020 con 0x7fe588004690 2023-10-18T16:12:43.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.649+0000 7fe59959a700 1 -- 172.21.15.88:0/1155560845 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe584007180 con 0x7fe588004690 2023-10-18T16:12:43.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.649+0000 7fe59959a700 1 -- 172.21.15.88:0/1155560845 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe584007540 con 0x7fe588004690 2023-10-18T16:12:43.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.650+0000 7fe5a0bfa700 1 -- 172.21.15.88:0/1155560845 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe588004690 msgr2=0x7fe588004b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:43.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.650+0000 7fe5a0bfa700 1 --2- 172.21.15.88:0/1155560845 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe588004690 0x7fe588004b10 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7fe584001700 tx=0x7fe584007bb0).stop 2023-10-18T16:12:43.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.650+0000 7fe5a0bfa700 1 -- 172.21.15.88:0/1155560845 shutdown_connections 2023-10-18T16:12:43.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.650+0000 7fe5a0bfa700 1 --2- 172.21.15.88:0/1155560845 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe588004690 0x7fe588004b10 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:43.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.650+0000 7fe5a0bfa700 1 -- 172.21.15.88:0/1155560845 >> 172.21.15.88:0/1155560845 conn(0x7fe59c06a810 msgr2=0x7fe59c0c6a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:12:43.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.650+0000 7fe5a0bfa700 1 -- 172.21.15.88:0/1155560845 shutdown_connections 2023-10-18T16:12:43.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.650+0000 7fe5a0bfa700 1 -- 172.21.15.88:0/1155560845 wait complete. 2023-10-18T16:12:43.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.651+0000 7fe5a0bfa700 1 Processor -- start 2023-10-18T16:12:43.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.651+0000 7fe5a0bfa700 1 -- start start 2023-10-18T16:12:43.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.651+0000 7fe5a0bfa700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe588004690 0x7fe59c0c63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:43.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.652+0000 7fe5a0bfa700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe59c150ef0 0x7fe59c0bcba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:43.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.652+0000 7fe5a0bfa700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe59c0bd090 0x7fe59c1602f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:43.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.652+0000 7fe5a0bfa700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe59c14c770 con 0x7fe588004690 2023-10-18T16:12:43.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.652+0000 7fe5a0bfa700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe59c0c68d0 con 0x7fe59c0bd090 2023-10-18T16:12:43.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.652+0000 7fe5a0bfa700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe59c15fe30 con 0x7fe59c150ef0 2023-10-18T16:12:43.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.652+0000 7fe59a59c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe588004690 0x7fe59c0c63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:43.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.652+0000 7fe59a59c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe588004690 0x7fe59c0c63e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:52826/0 (socket says 172.21.15.88:52826) 2023-10-18T16:12:43.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.652+0000 7fe59a59c700 1 -- 172.21.15.88:0/2198411963 learned_addr learned my addr 172.21.15.88:0/2198411963 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:12:43.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.652+0000 7fe599d9b700 1 --2- 172.21.15.88:0/2198411963 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe59c150ef0 0x7fe59c0bcba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:43.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.652+0000 7fe59ad9d700 1 --2- 172.21.15.88:0/2198411963 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe59c0bd090 0x7fe59c1602f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:43.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.652+0000 7fe59a59c700 1 -- 172.21.15.88:0/2198411963 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe59c150ef0 msgr2=0x7fe59c0bcba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:43.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.652+0000 7fe59a59c700 1 --2- 172.21.15.88:0/2198411963 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe59c150ef0 0x7fe59c0bcba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:43.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.652+0000 7fe59a59c700 1 -- 172.21.15.88:0/2198411963 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe59c0bd090 msgr2=0x7fe59c1602f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:43.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.652+0000 7fe59a59c700 1 --2- 172.21.15.88:0/2198411963 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe59c0bd090 0x7fe59c1602f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:43.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.652+0000 7fe59a59c700 1 -- 172.21.15.88:0/2198411963 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe59c14de40 con 0x7fe588004690 2023-10-18T16:12:43.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.652+0000 7fe59a59c700 1 --2- 172.21.15.88:0/2198411963 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe588004690 0x7fe59c0c63e0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7fe594011fb0 tx=0x7fe59400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:12:43.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.652+0000 7fe5927fc700 1 -- 172.21.15.88:0/2198411963 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe59400e040 con 0x7fe588004690 2023-10-18T16:12:43.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.652+0000 7fe5927fc700 1 -- 172.21.15.88:0/2198411963 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe59400f040 con 0x7fe588004690 2023-10-18T16:12:43.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.652+0000 7fe5a0bfa700 1 -- 172.21.15.88:0/2198411963 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe59c14e120 con 0x7fe588004690 2023-10-18T16:12:43.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.652+0000 7fe5927fc700 1 -- 172.21.15.88:0/2198411963 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe594015450 con 0x7fe588004690 2023-10-18T16:12:43.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.652+0000 7fe5a0bfa700 1 -- 172.21.15.88:0/2198411963 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe59c14e6e0 con 0x7fe588004690 2023-10-18T16:12:43.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.653+0000 7fe5a0bfa700 1 -- 172.21.15.88:0/2198411963 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe59c06c4e0 con 0x7fe588004690 2023-10-18T16:12:43.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.654+0000 7fe5927fc700 1 -- 172.21.15.88:0/2198411963 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7fe594015880 con 0x7fe588004690 2023-10-18T16:12:43.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.654+0000 7fe5927fc700 1 --2- 172.21.15.88:0/2198411963 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fe57c05f150 0x7fe57c061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:43.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.654+0000 7fe5927fc700 1 -- 172.21.15.88:0/2198411963 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6725+0+0 (secure 0 0 0) 0x7fe59407b400 con 0x7fe588004690 2023-10-18T16:12:43.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.654+0000 7fe599d9b700 1 --2- 172.21.15.88:0/2198411963 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fe57c05f150 0x7fe57c061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:43.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.655+0000 7fe599d9b700 1 --2- 172.21.15.88:0/2198411963 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fe57c05f150 0x7fe57c061610 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7fe584001010 tx=0x7fe584016000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:12:43.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.658+0000 7fe5927fc700 1 -- 172.21.15.88:0/2198411963 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7fe59404c470 con 0x7fe588004690 2023-10-18T16:12:43.865 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.862+0000 7fe5a0bfa700 1 -- 172.21.15.88:0/2198411963 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe59c065140 con 0x7fe588004690 2023-10-18T16:12:43.866 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.862+0000 7fe5927fc700 1 -- 172.21.15.88:0/2198411963 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7fe59404fa90 con 0x7fe588004690 2023-10-18T16:12:43.866 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:12:43.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.864+0000 7fe5a0bfa700 1 -- 172.21.15.88:0/2198411963 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fe57c05f150 msgr2=0x7fe57c061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:43.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.864+0000 7fe5a0bfa700 1 --2- 172.21.15.88:0/2198411963 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fe57c05f150 0x7fe57c061610 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7fe584001010 tx=0x7fe584016000).stop 2023-10-18T16:12:43.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.864+0000 7fe5a0bfa700 1 -- 172.21.15.88:0/2198411963 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe588004690 msgr2=0x7fe59c0c63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:43.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.864+0000 7fe5a0bfa700 1 --2- 172.21.15.88:0/2198411963 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe588004690 0x7fe59c0c63e0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7fe594011fb0 tx=0x7fe59400b040).stop 2023-10-18T16:12:43.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.864+0000 7fe5a0bfa700 1 -- 172.21.15.88:0/2198411963 shutdown_connections 2023-10-18T16:12:43.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.864+0000 7fe5a0bfa700 1 --2- 172.21.15.88:0/2198411963 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7fe57c05f150 0x7fe57c061610 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:43.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.864+0000 7fe5a0bfa700 1 --2- 172.21.15.88:0/2198411963 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe59c0bd090 0x7fe59c1602f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:43.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.864+0000 7fe5a0bfa700 1 --2- 172.21.15.88:0/2198411963 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe588004690 0x7fe59c0c63e0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:43.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.864+0000 7fe5a0bfa700 1 --2- 172.21.15.88:0/2198411963 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe59c150ef0 0x7fe59c0bcba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:43.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.864+0000 7fe5a0bfa700 1 -- 172.21.15.88:0/2198411963 >> 172.21.15.88:0/2198411963 conn(0x7fe59c06a810 msgr2=0x7fe59c151810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:12:43.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.864+0000 7fe5a0bfa700 1 -- 172.21.15.88:0/2198411963 shutdown_connections 2023-10-18T16:12:43.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:43.864+0000 7fe5a0bfa700 1 -- 172.21.15.88:0/2198411963 wait complete. 2023-10-18T16:12:43.879 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:12:45.224 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~189s 2023-10-18T16:12:45.225 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~202s 2023-10-18T16:12:45.225 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~235s 2023-10-18T16:12:45.225 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~215s 2023-10-18T16:12:45.226 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~296s 2023-10-18T16:12:45.226 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.f is failed for ~248s 2023-10-18T16:12:46.880 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:12:47.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.001+0000 7f7aed865700 1 Processor -- start 2023-10-18T16:12:47.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.001+0000 7f7aed865700 1 -- start start 2023-10-18T16:12:47.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.001+0000 7f7aed865700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7ae814d8c0 0x7f7ae8064440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:47.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.002+0000 7f7aed865700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f7ae80a9050 0x7f7ae8064930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:47.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.002+0000 7f7aed865700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ae8064150 con 0x7f7ae8065360 2023-10-18T16:12:47.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.002+0000 7f7aed865700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ae8153a40 con 0x7f7ae814d8c0 2023-10-18T16:12:47.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.002+0000 7f7aed865700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ae8063e70 con 0x7f7ae80a9050 2023-10-18T16:12:47.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.002+0000 7f7ae6ffd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7ae814d8c0 0x7f7ae8064440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:47.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.002+0000 7f7ae6ffd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7ae814d8c0 0x7f7ae8064440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:52836/0 (socket says 172.21.15.88:52836) 2023-10-18T16:12:47.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.002+0000 7f7ae6ffd700 1 -- 172.21.15.88:0/1458887874 learned_addr learned my addr 172.21.15.88:0/1458887874 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:12:47.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.002+0000 7f7ae67fc700 1 --2- 172.21.15.88:0/1458887874 >> v2:172.21.15.151:3300/0 conn(0x7f7ae80a9050 0x7f7ae8064930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:47.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.002+0000 7f7ae6ffd700 1 -- 172.21.15.88:0/1458887874 >> v2:172.21.15.151:3300/0 conn(0x7f7ae80a9050 msgr2=0x7f7ae8064930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:12:47.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.002+0000 7f7ae6ffd700 1 --2- 172.21.15.88:0/1458887874 >> v2:172.21.15.151:3300/0 conn(0x7f7ae80a9050 0x7f7ae8064930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:47.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.002+0000 7f7ae6ffd700 1 -- 172.21.15.88:0/1458887874 >> v1:172.21.15.151:6789/0 conn(0x7f7ae8065360 legacy=0x7f7ae8153ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:47.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.002+0000 7f7ae6ffd700 1 -- 172.21.15.88:0/1458887874 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ae80c72c0 con 0x7f7ae814d8c0 2023-10-18T16:12:47.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.002+0000 7f7ae6ffd700 1 --2- 172.21.15.88:0/1458887874 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7ae814d8c0 0x7f7ae8064440 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7f7ae80637d0 tx=0x7f7ad80090f0).ready entity=mon.0 client_cookie=634ad8a3807ff7db server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:12:47.010 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.002+0000 7f7ae5ffb700 1 -- 172.21.15.88:0/1458887874 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad800b520 con 0x7f7ae814d8c0 2023-10-18T16:12:47.010 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.002+0000 7f7ae5ffb700 1 -- 172.21.15.88:0/1458887874 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ad800b680 con 0x7f7ae814d8c0 2023-10-18T16:12:47.010 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.002+0000 7f7ae5ffb700 1 -- 172.21.15.88:0/1458887874 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad800ba20 con 0x7f7ae814d8c0 2023-10-18T16:12:47.010 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.003+0000 7f7aed865700 1 -- 172.21.15.88:0/1458887874 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7ae814d8c0 msgr2=0x7f7ae8064440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:47.010 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.003+0000 7f7aed865700 1 --2- 172.21.15.88:0/1458887874 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7ae814d8c0 0x7f7ae8064440 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7f7ae80637d0 tx=0x7f7ad80090f0).stop 2023-10-18T16:12:47.011 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.003+0000 7f7aed865700 1 -- 172.21.15.88:0/1458887874 shutdown_connections 2023-10-18T16:12:47.011 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.003+0000 7f7aed865700 1 --2- 172.21.15.88:0/1458887874 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7ae814d8c0 0x7f7ae8064440 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:47.011 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.003+0000 7f7aed865700 1 --2- 172.21.15.88:0/1458887874 >> v2:172.21.15.151:3300/0 conn(0x7f7ae80a9050 0x7f7ae8064930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:47.011 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.003+0000 7f7aed865700 1 -- 172.21.15.88:0/1458887874 >> 172.21.15.88:0/1458887874 conn(0x7f7ae806a810 msgr2=0x7f7ae80c6a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:12:47.011 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.003+0000 7f7aed865700 1 -- 172.21.15.88:0/1458887874 shutdown_connections 2023-10-18T16:12:47.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.003+0000 7f7aed865700 1 -- 172.21.15.88:0/1458887874 wait complete. 2023-10-18T16:12:47.014 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.004+0000 7f7aed865700 1 Processor -- start 2023-10-18T16:12:47.014 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.004+0000 7f7aed865700 1 -- start start 2023-10-18T16:12:47.015 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.005+0000 7f7aed865700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7ae8065360 0x7f7ae814f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:47.015 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.005+0000 7f7aed865700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7ae80a9050 0x7f7ae80bcf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:47.015 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.005+0000 7f7aed865700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7ae814d8c0 0x7f7ae8160cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:47.015 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.005+0000 7f7aed865700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae81605a0 con 0x7f7ae8065360 2023-10-18T16:12:47.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.005+0000 7f7aed865700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae80a3c40 con 0x7f7ae814d8c0 2023-10-18T16:12:47.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.005+0000 7f7aed865700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae80a42c0 con 0x7f7ae80a9050 2023-10-18T16:12:47.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.005+0000 7f7ae6ffd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7ae8065360 0x7f7ae814f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:47.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.005+0000 7f7ae6ffd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7ae8065360 0x7f7ae814f500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:52840/0 (socket says 172.21.15.88:52840) 2023-10-18T16:12:47.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.005+0000 7f7ae6ffd700 1 -- 172.21.15.88:0/3321223275 learned_addr learned my addr 172.21.15.88:0/3321223275 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:12:47.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.005+0000 7f7ae77fe700 1 --2- 172.21.15.88:0/3321223275 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7ae814d8c0 0x7f7ae8160cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:47.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.005+0000 7f7ae67fc700 1 --2- 172.21.15.88:0/3321223275 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7ae80a9050 0x7f7ae80bcf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:47.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.005+0000 7f7ae6ffd700 1 -- 172.21.15.88:0/3321223275 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7ae80a9050 msgr2=0x7f7ae80bcf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:47.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.005+0000 7f7ae6ffd700 1 --2- 172.21.15.88:0/3321223275 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7ae80a9050 0x7f7ae80bcf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:47.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.005+0000 7f7ae6ffd700 1 -- 172.21.15.88:0/3321223275 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7ae814d8c0 msgr2=0x7f7ae8160cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:47.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.005+0000 7f7ae6ffd700 1 --2- 172.21.15.88:0/3321223275 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7ae814d8c0 0x7f7ae8160cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:47.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.005+0000 7f7ae6ffd700 1 -- 172.21.15.88:0/3321223275 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ad8009370 con 0x7f7ae8065360 2023-10-18T16:12:47.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.005+0000 7f7ae77fe700 1 --2- 172.21.15.88:0/3321223275 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7ae814d8c0 0x7f7ae8160cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:12:47.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.005+0000 7f7ae67fc700 1 --2- 172.21.15.88:0/3321223275 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7ae80a9050 0x7f7ae80bcf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-18T16:12:47.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.005+0000 7f7ae6ffd700 1 --2- 172.21.15.88:0/3321223275 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7ae8065360 0x7f7ae814f500 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f7ad8006fd0 tx=0x7f7ad8017000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:12:47.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.005+0000 7f7aceffd700 1 -- 172.21.15.88:0/3321223275 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad800beb0 con 0x7f7ae8065360 2023-10-18T16:12:47.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.005+0000 7f7aceffd700 1 -- 172.21.15.88:0/3321223275 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ad800b7f0 con 0x7f7ae8065360 2023-10-18T16:12:47.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.005+0000 7f7aed865700 1 -- 172.21.15.88:0/3321223275 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ae814e580 con 0x7f7ae8065360 2023-10-18T16:12:47.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.005+0000 7f7aceffd700 1 -- 172.21.15.88:0/3321223275 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad801c420 con 0x7f7ae8065360 2023-10-18T16:12:47.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.005+0000 7f7aed865700 1 -- 172.21.15.88:0/3321223275 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ae814eb00 con 0x7f7ae8065360 2023-10-18T16:12:47.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.007+0000 7f7aed865700 1 -- 172.21.15.88:0/3321223275 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ae8160b20 con 0x7f7ae8065360 2023-10-18T16:12:47.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.007+0000 7f7aceffd700 1 -- 172.21.15.88:0/3321223275 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f7ad801c7d0 con 0x7f7ae8065360 2023-10-18T16:12:47.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.007+0000 7f7aceffd700 1 --2- 172.21.15.88:0/3321223275 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f7ad405f150 0x7f7ad4061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:47.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.007+0000 7f7aceffd700 1 -- 172.21.15.88:0/3321223275 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6725+0+0 (secure 0 0 0) 0x7f7ad807c040 con 0x7f7ae8065360 2023-10-18T16:12:47.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.007+0000 7f7ae67fc700 1 --2- 172.21.15.88:0/3321223275 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f7ad405f150 0x7f7ad4061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:47.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.008+0000 7f7ae67fc700 1 --2- 172.21.15.88:0/3321223275 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f7ad405f150 0x7f7ad4061610 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f7ae814d340 tx=0x7f7adc008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:12:47.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.011+0000 7f7aceffd700 1 -- 172.21.15.88:0/3321223275 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f7ad804d0b0 con 0x7f7ae8065360 2023-10-18T16:12:47.216 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.213+0000 7f7aed865700 1 -- 172.21.15.88:0/3321223275 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7ae80c75a0 con 0x7f7ae8065360 2023-10-18T16:12:47.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.213+0000 7f7aceffd700 1 -- 172.21.15.88:0/3321223275 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7f7ad802a070 con 0x7f7ae8065360 2023-10-18T16:12:47.217 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:12:47.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.215+0000 7f7aed865700 1 -- 172.21.15.88:0/3321223275 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f7ad405f150 msgr2=0x7f7ad4061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:47.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.215+0000 7f7aed865700 1 --2- 172.21.15.88:0/3321223275 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f7ad405f150 0x7f7ad4061610 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f7ae814d340 tx=0x7f7adc008040).stop 2023-10-18T16:12:47.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.215+0000 7f7aed865700 1 -- 172.21.15.88:0/3321223275 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7ae8065360 msgr2=0x7f7ae814f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:47.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.215+0000 7f7aed865700 1 --2- 172.21.15.88:0/3321223275 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7ae8065360 0x7f7ae814f500 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f7ad8006fd0 tx=0x7f7ad8017000).stop 2023-10-18T16:12:47.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.215+0000 7f7aed865700 1 -- 172.21.15.88:0/3321223275 shutdown_connections 2023-10-18T16:12:47.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.215+0000 7f7aed865700 1 --2- 172.21.15.88:0/3321223275 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f7ad405f150 0x7f7ad4061610 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:47.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.215+0000 7f7aed865700 1 --2- 172.21.15.88:0/3321223275 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7ae814d8c0 0x7f7ae8160cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:47.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.215+0000 7f7aed865700 1 --2- 172.21.15.88:0/3321223275 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7ae8065360 0x7f7ae814f500 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:47.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.215+0000 7f7aed865700 1 --2- 172.21.15.88:0/3321223275 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7ae80a9050 0x7f7ae80bcf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:47.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.215+0000 7f7aed865700 1 -- 172.21.15.88:0/3321223275 >> 172.21.15.88:0/3321223275 conn(0x7f7ae806a810 msgr2=0x7f7ae81526a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:12:47.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.215+0000 7f7aed865700 1 -- 172.21.15.88:0/3321223275 shutdown_connections 2023-10-18T16:12:47.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:47.215+0000 7f7aed865700 1 -- 172.21.15.88:0/3321223275 wait complete. 2023-10-18T16:12:47.230 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:12:50.232 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:12:50.355 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.352+0000 7f2bcf292700 1 Processor -- start 2023-10-18T16:12:50.356 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.352+0000 7f2bcf292700 1 -- start start 2023-10-18T16:12:50.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.353+0000 7f2bcf292700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2bc814d8c0 0x7f2bc8064440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:50.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.353+0000 7f2bcf292700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f2bc80a9050 0x7f2bc8064930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:50.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.353+0000 7f2bcf292700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bc8064150 con 0x7f2bc8065360 2023-10-18T16:12:50.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.353+0000 7f2bcf292700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bc8153a40 con 0x7f2bc814d8c0 2023-10-18T16:12:50.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.353+0000 7f2bcf292700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bc8063e70 con 0x7f2bc80a9050 2023-10-18T16:12:50.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.353+0000 7f2bcd02e700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2bc814d8c0 0x7f2bc8064440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:50.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.353+0000 7f2bcd82f700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f2bc8065360 0x7f2bc8153ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:58246/0 (socket says 172.21.15.88:58246) 2023-10-18T16:12:50.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.353+0000 7f2bcd82f700 1 -- 172.21.15.88:0/1519947326 learned_addr learned my addr 172.21.15.88:0/1519947326 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-18T16:12:50.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.353+0000 7f2bcc82d700 1 --2- 172.21.15.88:0/1519947326 >> v2:172.21.15.151:3300/0 conn(0x7f2bc80a9050 0x7f2bc8064930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:50.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.353+0000 7f2bcd02e700 1 -- 172.21.15.88:0/1519947326 >> v2:172.21.15.151:3300/0 conn(0x7f2bc80a9050 msgr2=0x7f2bc8064930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-18T16:12:50.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.353+0000 7f2bcd02e700 1 --2- 172.21.15.88:0/1519947326 >> v2:172.21.15.151:3300/0 conn(0x7f2bc80a9050 0x7f2bc8064930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:50.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.353+0000 7f2bcd02e700 1 -- 172.21.15.88:0/1519947326 >> v1:172.21.15.88:6789/0 conn(0x7f2bc8065360 legacy=0x7f2bc8153ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:50.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.353+0000 7f2bbffff700 1 -- 172.21.15.88:0/1519947326 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2134754795 0 0) 0x7f2bc8064150 con 0x7f2bc8065360 2023-10-18T16:12:50.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.353+0000 7f2bcd02e700 1 -- 172.21.15.88:0/1519947326 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bc80c72c0 con 0x7f2bc814d8c0 2023-10-18T16:12:50.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.353+0000 7f2bcd02e700 1 --2- 172.21.15.88:0/1519947326 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2bc814d8c0 0x7f2bc8064440 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f2bc80637d0 tx=0x7f2bc4009060).ready entity=mon.0 client_cookie=c33d350741d741c7 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:12:50.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.353+0000 7f2bbffff700 1 -- 172.21.15.88:0/1519947326 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc40090d0 con 0x7f2bc814d8c0 2023-10-18T16:12:50.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.353+0000 7f2bbffff700 1 -- 172.21.15.88:0/1519947326 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bc400f510 con 0x7f2bc814d8c0 2023-10-18T16:12:50.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.353+0000 7f2bbffff700 1 -- 172.21.15.88:0/1519947326 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc400f890 con 0x7f2bc814d8c0 2023-10-18T16:12:50.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.354+0000 7f2bcf292700 1 -- 172.21.15.88:0/1519947326 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2bc814d8c0 msgr2=0x7f2bc8064440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:50.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.354+0000 7f2bcf292700 1 --2- 172.21.15.88:0/1519947326 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2bc814d8c0 0x7f2bc8064440 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f2bc80637d0 tx=0x7f2bc4009060).stop 2023-10-18T16:12:50.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.354+0000 7f2bcf292700 1 -- 172.21.15.88:0/1519947326 shutdown_connections 2023-10-18T16:12:50.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.354+0000 7f2bcf292700 1 --2- 172.21.15.88:0/1519947326 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2bc814d8c0 0x7f2bc8064440 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:50.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.354+0000 7f2bcf292700 1 --2- 172.21.15.88:0/1519947326 >> v2:172.21.15.151:3300/0 conn(0x7f2bc80a9050 0x7f2bc8064930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:50.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.354+0000 7f2bcf292700 1 -- 172.21.15.88:0/1519947326 >> 172.21.15.88:0/1519947326 conn(0x7f2bc806a810 msgr2=0x7f2bc80c6a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:12:50.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.354+0000 7f2bcf292700 1 -- 172.21.15.88:0/1519947326 shutdown_connections 2023-10-18T16:12:50.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.354+0000 7f2bcf292700 1 -- 172.21.15.88:0/1519947326 wait complete. 2023-10-18T16:12:50.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.355+0000 7f2bcf292700 1 Processor -- start 2023-10-18T16:12:50.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.355+0000 7f2bcf292700 1 -- start start 2023-10-18T16:12:50.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.356+0000 7f2bcf292700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2bc8065360 0x7f2bc8150e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:50.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.356+0000 7f2bcf292700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2bc80a9050 0x7f2bc8160a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:50.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.356+0000 7f2bcf292700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2bc814d8c0 0x7f2bc814f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:50.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.356+0000 7f2bcf292700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bc80c26a0 con 0x7f2bc8065360 2023-10-18T16:12:50.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.356+0000 7f2bcf292700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bc814ef10 con 0x7f2bc814d8c0 2023-10-18T16:12:50.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.356+0000 7f2bcf292700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bc80aba40 con 0x7f2bc80a9050 2023-10-18T16:12:50.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.356+0000 7f2bcd02e700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2bc8065360 0x7f2bc8150e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:50.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.356+0000 7f2bcd02e700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2bc8065360 0x7f2bc8150e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42726/0 (socket says 172.21.15.88:42726) 2023-10-18T16:12:50.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.356+0000 7f2bcd02e700 1 -- 172.21.15.88:0/3586406489 learned_addr learned my addr 172.21.15.88:0/3586406489 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-18T16:12:50.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.356+0000 7f2bcd82f700 1 --2- 172.21.15.88:0/3586406489 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2bc814d8c0 0x7f2bc814f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:50.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.356+0000 7f2bcc82d700 1 --2- 172.21.15.88:0/3586406489 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2bc80a9050 0x7f2bc8160a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:50.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.356+0000 7f2bcd02e700 1 -- 172.21.15.88:0/3586406489 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2bc80a9050 msgr2=0x7f2bc8160a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:50.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.356+0000 7f2bcd02e700 1 --2- 172.21.15.88:0/3586406489 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2bc80a9050 0x7f2bc8160a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:50.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.356+0000 7f2bcd02e700 1 -- 172.21.15.88:0/3586406489 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2bc814d8c0 msgr2=0x7f2bc814f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:50.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.357+0000 7f2bcd02e700 1 --2- 172.21.15.88:0/3586406489 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2bc814d8c0 0x7f2bc814f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:50.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.357+0000 7f2bcd02e700 1 -- 172.21.15.88:0/3586406489 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bc40092a0 con 0x7f2bc8065360 2023-10-18T16:12:50.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.357+0000 7f2bcd82f700 1 --2- 172.21.15.88:0/3586406489 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2bc814d8c0 0x7f2bc814f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-10-18T16:12:50.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.357+0000 7f2bcc82d700 1 --2- 172.21.15.88:0/3586406489 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2bc80a9050 0x7f2bc8160a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-10-18T16:12:50.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.357+0000 7f2bcd02e700 1 --2- 172.21.15.88:0/3586406489 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2bc8065360 0x7f2bc8150e40 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7f2bc4009dd0 tx=0x7f2bc4017000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:12:50.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.357+0000 7f2bbd7fa700 1 -- 172.21.15.88:0/3586406489 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc4006950 con 0x7f2bc8065360 2023-10-18T16:12:50.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.357+0000 7f2bbd7fa700 1 -- 172.21.15.88:0/3586406489 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bc400fc50 con 0x7f2bc8065360 2023-10-18T16:12:50.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.357+0000 7f2bcf292700 1 -- 172.21.15.88:0/3586406489 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bc814ebc0 con 0x7f2bc8065360 2023-10-18T16:12:50.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.357+0000 7f2bbd7fa700 1 -- 172.21.15.88:0/3586406489 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc401b440 con 0x7f2bc8065360 2023-10-18T16:12:50.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.357+0000 7f2bcf292700 1 -- 172.21.15.88:0/3586406489 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bc80abfe0 con 0x7f2bc8065360 2023-10-18T16:12:50.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.358+0000 7f2bcf292700 1 -- 172.21.15.88:0/3586406489 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bc806c4e0 con 0x7f2bc8065360 2023-10-18T16:12:50.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.358+0000 7f2bbd7fa700 1 -- 172.21.15.88:0/3586406489 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 75446+0+0 (secure 0 0 0) 0x7f2bc401b830 con 0x7f2bc8065360 2023-10-18T16:12:50.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.358+0000 7f2bbd7fa700 1 --2- 172.21.15.88:0/3586406489 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f2bac05f150 0x7f2bac061610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:50.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.359+0000 7f2bbd7fa700 1 -- 172.21.15.88:0/3586406489 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6725+0+0 (secure 0 0 0) 0x7f2bc407bab0 con 0x7f2bc8065360 2023-10-18T16:12:50.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.359+0000 7f2bcc82d700 1 --2- 172.21.15.88:0/3586406489 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f2bac05f150 0x7f2bac061610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-18T16:12:50.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.359+0000 7f2bcc82d700 1 --2- 172.21.15.88:0/3586406489 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f2bac05f150 0x7f2bac061610 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f2bb8006d80 tx=0x7f2bb8008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-18T16:12:50.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.363+0000 7f2bbd7fa700 1 -- 172.21.15.88:0/3586406489 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162466 (secure 0 0 0) 0x7f2bc404cb20 con 0x7f2bc8065360 2023-10-18T16:12:50.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.565+0000 7f2bcf292700 1 -- 172.21.15.88:0/3586406489 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2bc814f1f0 con 0x7f2bc8065360 2023-10-18T16:12:50.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.565+0000 7f2bbd7fa700 1 -- 172.21.15.88:0/3586406489 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+195 (secure 0 0 0) 0x7f2bc4050140 con 0x7f2bc8065360 2023-10-18T16:12:50.570 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-18T16:12:50.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.567+0000 7f2bcf292700 1 -- 172.21.15.88:0/3586406489 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f2bac05f150 msgr2=0x7f2bac061610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:50.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.567+0000 7f2bcf292700 1 --2- 172.21.15.88:0/3586406489 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f2bac05f150 0x7f2bac061610 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f2bb8006d80 tx=0x7f2bb8008040).stop 2023-10-18T16:12:50.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.567+0000 7f2bcf292700 1 -- 172.21.15.88:0/3586406489 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2bc8065360 msgr2=0x7f2bc8150e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-18T16:12:50.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.567+0000 7f2bcf292700 1 --2- 172.21.15.88:0/3586406489 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2bc8065360 0x7f2bc8150e40 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7f2bc4009dd0 tx=0x7f2bc4017000).stop 2023-10-18T16:12:50.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.567+0000 7f2bcf292700 1 -- 172.21.15.88:0/3586406489 shutdown_connections 2023-10-18T16:12:50.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.567+0000 7f2bcf292700 1 --2- 172.21.15.88:0/3586406489 >> [v2:172.21.15.151:6832/90916,v1:172.21.15.151:6833/90916] conn(0x7f2bac05f150 0x7f2bac061610 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:50.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.567+0000 7f2bcf292700 1 --2- 172.21.15.88:0/3586406489 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2bc814d8c0 0x7f2bc814f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:50.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.567+0000 7f2bcf292700 1 --2- 172.21.15.88:0/3586406489 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2bc8065360 0x7f2bc8150e40 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:50.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.567+0000 7f2bcf292700 1 --2- 172.21.15.88:0/3586406489 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2bc80a9050 0x7f2bc8160a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-18T16:12:50.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.567+0000 7f2bcf292700 1 -- 172.21.15.88:0/3586406489 >> 172.21.15.88:0/3586406489 conn(0x7f2bc806a810 msgr2=0x7f2bc8153ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-18T16:12:50.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.567+0000 7f2bcf292700 1 -- 172.21.15.88:0/3586406489 shutdown_connections 2023-10-18T16:12:50.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:50.567+0000 7f2bcf292700 1 -- 172.21.15.88:0/3586406489 wait complete. 2023-10-18T16:12:50.583 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-18T16:12:51.537 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~196s 2023-10-18T16:12:51.538 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~208s 2023-10-18T16:12:51.538 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~241s 2023-10-18T16:12:51.538 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~222s 2023-10-18T16:12:51.538 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~302s 2023-10-18T16:12:51.539 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.f is failed for ~255s 2023-10-18T16:12:51.539 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2023-10-18T16:12:52.846 INFO:tasks.ceph.osd.0:Sent signal 15 2023-10-18T16:12:52.847 INFO:tasks.ceph.osd.1:Sent signal 15 2023-10-18T16:12:52.848 INFO:tasks.ceph.osd.2:Sent signal 15 2023-10-18T16:12:52.848 INFO:tasks.ceph.osd.3:Sent signal 15 2023-10-18T16:12:52.848 INFO:tasks.ceph.osd.4:Sent signal 15 2023-10-18T16:12:52.849 INFO:tasks.ceph.osd.5:Sent signal 15 2023-10-18T16:12:52.849 INFO:tasks.ceph.osd.6:Sent signal 15 2023-10-18T16:12:52.850 INFO:tasks.ceph.osd.7:Sent signal 15 2023-10-18T16:12:52.850 INFO:tasks.ceph.mon.a:Sent signal 15 2023-10-18T16:12:52.850 INFO:tasks.ceph.mon.b:Sent signal 15 2023-10-18T16:12:52.851 INFO:tasks.ceph.mon.c:Sent signal 15 2023-10-18T16:12:52.851 INFO:tasks.ceph.mgr.y:Sent signal 15 2023-10-18T16:12:52.851 INFO:tasks.ceph.mgr.x:Sent signal 15 2023-10-18T16:12:52.852 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-10-18T16:12:52.844+0000 7f955445c700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 90976) UID: 0 2023-10-18T16:12:52.853 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-10-18T16:12:52.844+0000 7f955445c700 -1 osd.0 29 *** Got signal Terminated *** 2023-10-18T16:12:52.853 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-10-18T16:12:52.844+0000 7f955445c700 -1 osd.0 29 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-10-18T16:12:52.853 INFO:tasks.ceph.osd.4.smithi151.stderr:2023-10-18T16:12:52.846+0000 7f1d38cc6700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 91503) UID: 0 2023-10-18T16:12:52.854 INFO:tasks.ceph.osd.4.smithi151.stderr:2023-10-18T16:12:52.846+0000 7f1d38cc6700 -1 osd.4 29 *** Got signal Terminated *** 2023-10-18T16:12:52.854 INFO:tasks.ceph.osd.4.smithi151.stderr:2023-10-18T16:12:52.846+0000 7f1d38cc6700 -1 osd.4 29 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-10-18T16:12:52.854 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-10-18T16:12:52.845+0000 7f25e0c93700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 90981) UID: 0 2023-10-18T16:12:52.855 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-10-18T16:12:52.845+0000 7f25e0c93700 -1 osd.1 29 *** Got signal Terminated *** 2023-10-18T16:12:52.855 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-10-18T16:12:52.845+0000 7f25e0c93700 -1 osd.1 29 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-10-18T16:12:52.855 INFO:tasks.ceph.osd.5.smithi151.stderr:2023-10-18T16:12:52.846+0000 7f3832941700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 91505) UID: 0 2023-10-18T16:12:52.855 INFO:tasks.ceph.osd.5.smithi151.stderr:2023-10-18T16:12:52.846+0000 7f3832941700 -1 osd.5 29 *** Got signal Terminated *** 2023-10-18T16:12:52.856 INFO:tasks.ceph.osd.5.smithi151.stderr:2023-10-18T16:12:52.846+0000 7f3832941700 -1 osd.5 29 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-10-18T16:12:52.856 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-10-18T16:12:52.845+0000 7f0a855a1700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 90988) UID: 0 2023-10-18T16:12:52.856 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-10-18T16:12:52.845+0000 7f0a855a1700 -1 osd.2 29 *** Got signal Terminated *** 2023-10-18T16:12:52.857 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-10-18T16:12:52.845+0000 7f0a855a1700 -1 osd.2 29 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-10-18T16:12:52.857 INFO:tasks.ceph.osd.6.smithi151.stderr:2023-10-18T16:12:52.846+0000 7f813d875700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 91508) UID: 0 2023-10-18T16:12:52.857 INFO:tasks.ceph.osd.6.smithi151.stderr:2023-10-18T16:12:52.846+0000 7f813d875700 -1 osd.6 29 *** Got signal Terminated *** 2023-10-18T16:12:52.857 INFO:tasks.ceph.osd.6.smithi151.stderr:2023-10-18T16:12:52.846+0000 7f813d875700 -1 osd.6 29 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-10-18T16:12:52.858 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-10-18T16:12:52.846+0000 7fd713538700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 90992) UID: 0 2023-10-18T16:12:52.858 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-10-18T16:12:52.846+0000 7fd713538700 -1 osd.3 29 *** Got signal Terminated *** 2023-10-18T16:12:52.858 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-10-18T16:12:52.846+0000 7fd713538700 -1 osd.3 29 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-10-18T16:12:52.858 INFO:tasks.ceph.osd.7.smithi151.stderr:2023-10-18T16:12:52.847+0000 7f11fcde8700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 91511) UID: 0 2023-10-18T16:12:52.859 INFO:tasks.ceph.osd.7.smithi151.stderr:2023-10-18T16:12:52.847+0000 7f11fcde8700 -1 osd.7 29 *** Got signal Terminated *** 2023-10-18T16:12:52.859 INFO:tasks.ceph.osd.7.smithi151.stderr:2023-10-18T16:12:52.847+0000 7f11fcde8700 -1 osd.7 29 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-10-18T16:12:52.859 INFO:tasks.ceph.mon.a.smithi088.stderr:2023-10-18T16:12:52.847+0000 7f5e6933f700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster ceph -i a (PID: 90569) UID: 0 2023-10-18T16:12:52.860 INFO:tasks.ceph.mon.a.smithi088.stderr:2023-10-18T16:12:52.847+0000 7f5e6933f700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-10-18T16:12:52.860 INFO:tasks.ceph.mon.b.smithi151.stderr:2023-10-18T16:12:52.848+0000 7fce9c8a4700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster ceph -i b (PID: 90906) UID: 0 2023-10-18T16:12:52.860 INFO:tasks.ceph.mon.b.smithi151.stderr:2023-10-18T16:12:52.848+0000 7fce9c8a4700 -1 mon.b@1(peon) e1 *** Got Signal Terminated *** 2023-10-18T16:12:52.861 INFO:tasks.ceph.mon.c.smithi151.stderr:2023-10-18T16:12:52.849+0000 7f262a3d2700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster ceph -i c (PID: 90909) UID: 0 2023-10-18T16:12:52.861 INFO:tasks.ceph.mon.c.smithi151.stderr:2023-10-18T16:12:52.849+0000 7f262a3d2700 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2023-10-18T16:12:52.861 INFO:tasks.ceph.mgr.x.smithi151.stderr:2023-10-18T16:12:52.849+0000 7f8d8c1a6700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster ceph -i x (PID: 90912) UID: 0 2023-10-18T16:12:52.861 INFO:tasks.ceph.mgr.x.smithi151.stderr:2023-10-18T16:12:52.849+0000 7f8d8c1a6700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2023-10-18T16:12:53.052 INFO:tasks.ceph.mgr.y.smithi088.stderr:daemon-helper: command crashed with signal 15 2023-10-18T16:12:53.091 INFO:tasks.ceph.mon.c.smithi151.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.14-100-gb4ddd52c/rpm/el8/BUILD/ceph-16.2.14-100-gb4ddd52c/src/mon/Monitor.cc: In function 'virtual Monitor::~Monitor()' thread 7f263a17f880 time 2023-10-18T16:12:53.089058+0000 2023-10-18T16:12:53.092 INFO:tasks.ceph.mon.c.smithi151.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.14-100-gb4ddd52c/rpm/el8/BUILD/ceph-16.2.14-100-gb4ddd52c/src/mon/Monitor.cc: 287: FAILED ceph_assert(session_map.sessions.empty()) 2023-10-18T16:12:53.092 INFO:tasks.ceph.mon.c.smithi151.stderr: ceph version 16.2.14-100-gb4ddd52c (b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5) pacific (stable) 2023-10-18T16:12:53.093 INFO:tasks.ceph.mon.c.smithi151.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f26312f2c14] 2023-10-18T16:12:53.093 INFO:tasks.ceph.mon.c.smithi151.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x279e2e) [0x7f26312f2e2e] 2023-10-18T16:12:53.093 INFO:tasks.ceph.mon.c.smithi151.stderr: 3: (Monitor::~Monitor()+0xe40) [0x55edf4386b80] 2023-10-18T16:12:53.093 INFO:tasks.ceph.mon.c.smithi151.stderr: 4: (Monitor::~Monitor()+0xd) [0x55edf4386bdd] 2023-10-18T16:12:53.093 INFO:tasks.ceph.mon.c.smithi151.stderr: 5: main() 2023-10-18T16:12:53.094 INFO:tasks.ceph.mon.c.smithi151.stderr: 6: __libc_start_main() 2023-10-18T16:12:53.094 INFO:tasks.ceph.mon.c.smithi151.stderr: 7: _start() 2023-10-18T16:12:53.094 INFO:tasks.ceph.mon.c.smithi151.stderr:*** Caught signal (Aborted) ** 2023-10-18T16:12:53.094 INFO:tasks.ceph.mon.c.smithi151.stderr: in thread 7f263a17f880 thread_name:ceph-mon 2023-10-18T16:12:53.095 INFO:tasks.ceph.mon.c.smithi151.stderr:2023-10-18T16:12:53.088+0000 7f263a17f880 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.14-100-gb4ddd52c/rpm/el8/BUILD/ceph-16.2.14-100-gb4ddd52c/src/mon/Monitor.cc: In function 'virtual Monitor::~Monitor()' thread 7f263a17f880 time 2023-10-18T16:12:53.089058+0000 2023-10-18T16:12:53.095 INFO:tasks.ceph.mon.c.smithi151.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.14-100-gb4ddd52c/rpm/el8/BUILD/ceph-16.2.14-100-gb4ddd52c/src/mon/Monitor.cc: 287: FAILED ceph_assert(session_map.sessions.empty()) 2023-10-18T16:12:53.095 INFO:tasks.ceph.mon.c.smithi151.stderr: 2023-10-18T16:12:53.095 INFO:tasks.ceph.mon.c.smithi151.stderr: ceph version 16.2.14-100-gb4ddd52c (b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5) pacific (stable) 2023-10-18T16:12:53.096 INFO:tasks.ceph.mon.c.smithi151.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f26312f2c14] 2023-10-18T16:12:53.096 INFO:tasks.ceph.mon.c.smithi151.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x279e2e) [0x7f26312f2e2e] 2023-10-18T16:12:53.096 INFO:tasks.ceph.mon.c.smithi151.stderr: 3: (Monitor::~Monitor()+0xe40) [0x55edf4386b80] 2023-10-18T16:12:53.096 INFO:tasks.ceph.mon.c.smithi151.stderr: 4: (Monitor::~Monitor()+0xd) [0x55edf4386bdd] 2023-10-18T16:12:53.096 INFO:tasks.ceph.mon.c.smithi151.stderr: 5: main() 2023-10-18T16:12:53.097 INFO:tasks.ceph.mon.c.smithi151.stderr: 6: __libc_start_main() 2023-10-18T16:12:53.097 INFO:tasks.ceph.mon.c.smithi151.stderr: 7: _start() 2023-10-18T16:12:53.097 INFO:tasks.ceph.mon.c.smithi151.stderr: 2023-10-18T16:12:53.097 INFO:tasks.ceph.mon.c.smithi151.stderr: ceph version 16.2.14-100-gb4ddd52c (b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5) pacific (stable) 2023-10-18T16:12:53.097 INFO:tasks.ceph.mon.c.smithi151.stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f262f02eb20] 2023-10-18T16:12:53.098 INFO:tasks.ceph.mon.c.smithi151.stderr: 2: gsignal() 2023-10-18T16:12:53.098 INFO:tasks.ceph.mon.c.smithi151.stderr: 3: abort() 2023-10-18T16:12:53.098 INFO:tasks.ceph.mon.c.smithi151.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f26312f2c65] 2023-10-18T16:12:53.098 INFO:tasks.ceph.mon.c.smithi151.stderr: 5: /usr/lib64/ceph/libceph-common.so.2(+0x279e2e) [0x7f26312f2e2e] 2023-10-18T16:12:53.098 INFO:tasks.ceph.mon.c.smithi151.stderr: 6: (Monitor::~Monitor()+0xe40) [0x55edf4386b80] 2023-10-18T16:12:53.099 INFO:tasks.ceph.mon.c.smithi151.stderr: 7: (Monitor::~Monitor()+0xd) [0x55edf4386bdd] 2023-10-18T16:12:53.099 INFO:tasks.ceph.mon.c.smithi151.stderr: 8: main() 2023-10-18T16:12:53.099 INFO:tasks.ceph.mon.c.smithi151.stderr: 9: __libc_start_main() 2023-10-18T16:12:53.099 INFO:tasks.ceph.mon.c.smithi151.stderr: 10: _start() 2023-10-18T16:12:53.100 INFO:tasks.ceph.mon.c.smithi151.stderr:2023-10-18T16:12:53.090+0000 7f263a17f880 -1 *** Caught signal (Aborted) ** 2023-10-18T16:12:53.100 INFO:tasks.ceph.mon.c.smithi151.stderr: in thread 7f263a17f880 thread_name:ceph-mon 2023-10-18T16:12:53.100 INFO:tasks.ceph.mon.c.smithi151.stderr: 2023-10-18T16:12:53.100 INFO:tasks.ceph.mon.c.smithi151.stderr: ceph version 16.2.14-100-gb4ddd52c (b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5) pacific (stable) 2023-10-18T16:12:53.100 INFO:tasks.ceph.mon.c.smithi151.stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f262f02eb20] 2023-10-18T16:12:53.101 INFO:tasks.ceph.mon.c.smithi151.stderr: 2: gsignal() 2023-10-18T16:12:53.101 INFO:tasks.ceph.mon.c.smithi151.stderr: 3: abort() 2023-10-18T16:12:53.101 INFO:tasks.ceph.mon.c.smithi151.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f26312f2c65] 2023-10-18T16:12:53.101 INFO:tasks.ceph.mon.c.smithi151.stderr: 5: /usr/lib64/ceph/libceph-common.so.2(+0x279e2e) [0x7f26312f2e2e] 2023-10-18T16:12:53.101 INFO:tasks.ceph.mon.c.smithi151.stderr: 6: (Monitor::~Monitor()+0xe40) [0x55edf4386b80] 2023-10-18T16:12:53.102 INFO:tasks.ceph.mon.c.smithi151.stderr: 7: (Monitor::~Monitor()+0xd) [0x55edf4386bdd] 2023-10-18T16:12:53.102 INFO:tasks.ceph.mon.c.smithi151.stderr: 8: main() 2023-10-18T16:12:53.102 INFO:tasks.ceph.mon.c.smithi151.stderr: 9: __libc_start_main() 2023-10-18T16:12:53.102 INFO:tasks.ceph.mon.c.smithi151.stderr: 10: _start() 2023-10-18T16:12:53.102 INFO:tasks.ceph.mon.c.smithi151.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-10-18T16:12:53.103 INFO:tasks.ceph.mon.c.smithi151.stderr: 2023-10-18T16:12:53.103 INFO:tasks.ceph.mon.c.smithi151.stderr: -117> 2023-10-18T16:12:52.849+0000 7f262a3d2700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster ceph -i c (PID: 90909) UID: 0 2023-10-18T16:12:53.104 INFO:tasks.ceph.mon.c.smithi151.stderr: -116> 2023-10-18T16:12:52.849+0000 7f262a3d2700 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2023-10-18T16:12:53.104 INFO:tasks.ceph.mon.c.smithi151.stderr: -1> 2023-10-18T16:12:53.088+0000 7f263a17f880 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.14-100-gb4ddd52c/rpm/el8/BUILD/ceph-16.2.14-100-gb4ddd52c/src/mon/Monitor.cc: In function 'virtual Monitor::~Monitor()' thread 7f263a17f880 time 2023-10-18T16:12:53.089058+0000 2023-10-18T16:12:53.104 INFO:tasks.ceph.mon.c.smithi151.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.14-100-gb4ddd52c/rpm/el8/BUILD/ceph-16.2.14-100-gb4ddd52c/src/mon/Monitor.cc: 287: FAILED ceph_assert(session_map.sessions.empty()) 2023-10-18T16:12:53.104 INFO:tasks.ceph.mon.c.smithi151.stderr: 2023-10-18T16:12:53.105 INFO:tasks.ceph.mon.c.smithi151.stderr: ceph version 16.2.14-100-gb4ddd52c (b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5) pacific (stable) 2023-10-18T16:12:53.105 INFO:tasks.ceph.mon.c.smithi151.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f26312f2c14] 2023-10-18T16:12:53.105 INFO:tasks.ceph.mon.c.smithi151.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x279e2e) [0x7f26312f2e2e] 2023-10-18T16:12:53.106 INFO:tasks.ceph.mon.c.smithi151.stderr: 3: (Monitor::~Monitor()+0xe40) [0x55edf4386b80] 2023-10-18T16:12:53.106 INFO:tasks.ceph.mon.c.smithi151.stderr: 4: (Monitor::~Monitor()+0xd) [0x55edf4386bdd] 2023-10-18T16:12:53.106 INFO:tasks.ceph.mon.c.smithi151.stderr: 5: main() 2023-10-18T16:12:53.106 INFO:tasks.ceph.mon.c.smithi151.stderr: 6: __libc_start_main() 2023-10-18T16:12:53.106 INFO:tasks.ceph.mon.c.smithi151.stderr: 7: _start() 2023-10-18T16:12:53.107 INFO:tasks.ceph.mon.c.smithi151.stderr: 2023-10-18T16:12:53.107 INFO:tasks.ceph.mon.c.smithi151.stderr: 0> 2023-10-18T16:12:53.090+0000 7f263a17f880 -1 *** Caught signal (Aborted) ** 2023-10-18T16:12:53.107 INFO:tasks.ceph.mon.c.smithi151.stderr: in thread 7f263a17f880 thread_name:ceph-mon 2023-10-18T16:12:53.107 INFO:tasks.ceph.mon.c.smithi151.stderr: 2023-10-18T16:12:53.107 INFO:tasks.ceph.mon.c.smithi151.stderr: ceph version 16.2.14-100-gb4ddd52c (b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5) pacific (stable) 2023-10-18T16:12:53.108 INFO:tasks.ceph.mon.c.smithi151.stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f262f02eb20] 2023-10-18T16:12:53.108 INFO:tasks.ceph.mon.c.smithi151.stderr: 2: gsignal() 2023-10-18T16:12:53.108 INFO:tasks.ceph.mon.c.smithi151.stderr: 3: abort() 2023-10-18T16:12:53.108 INFO:tasks.ceph.mon.c.smithi151.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f26312f2c65] 2023-10-18T16:12:53.108 INFO:tasks.ceph.mon.c.smithi151.stderr: 5: /usr/lib64/ceph/libceph-common.so.2(+0x279e2e) [0x7f26312f2e2e] 2023-10-18T16:12:53.109 INFO:tasks.ceph.mon.c.smithi151.stderr: 6: (Monitor::~Monitor()+0xe40) [0x55edf4386b80] 2023-10-18T16:12:53.109 INFO:tasks.ceph.mon.c.smithi151.stderr: 7: (Monitor::~Monitor()+0xd) [0x55edf4386bdd] 2023-10-18T16:12:53.109 INFO:tasks.ceph.mon.c.smithi151.stderr: 8: main() 2023-10-18T16:12:53.109 INFO:tasks.ceph.mon.c.smithi151.stderr: 9: __libc_start_main() 2023-10-18T16:12:53.109 INFO:tasks.ceph.mon.c.smithi151.stderr: 10: _start() 2023-10-18T16:12:53.110 INFO:tasks.ceph.mon.c.smithi151.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-10-18T16:12:53.110 INFO:tasks.ceph.mon.c.smithi151.stderr: 2023-10-18T16:12:53.110 INFO:tasks.ceph.mon.c.smithi151.stderr:*** Caught signal (Segmentation fault) ** 2023-10-18T16:12:53.110 INFO:tasks.ceph.mon.c.smithi151.stderr: in thread 7f263a17f880 thread_name:ceph-mon 2023-10-18T16:12:53.111 INFO:tasks.ceph.mon.c.smithi151.stderr: ceph version 16.2.14-100-gb4ddd52c (b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5) pacific (stable) 2023-10-18T16:12:53.111 INFO:tasks.ceph.mon.c.smithi151.stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f262f02eb20] 2023-10-18T16:12:53.111 INFO:tasks.ceph.mon.c.smithi151.stderr: 2: pthread_getname_np() 2023-10-18T16:12:53.111 INFO:tasks.ceph.mon.c.smithi151.stderr: 3: (ceph::logging::Log::dump_recent()+0x4c2) [0x7f2631678842] 2023-10-18T16:12:53.112 INFO:tasks.ceph.mon.c.smithi151.stderr: 4: ceph-mon(+0x54738b) [0x55edf45e638b] 2023-10-18T16:12:53.112 INFO:tasks.ceph.mon.c.smithi151.stderr: 5: /lib64/libpthread.so.0(+0x12b20) [0x7f262f02eb20] 2023-10-18T16:12:53.112 INFO:tasks.ceph.mon.c.smithi151.stderr: 6: gsignal() 2023-10-18T16:12:53.112 INFO:tasks.ceph.mon.c.smithi151.stderr: 7: abort() 2023-10-18T16:12:53.113 INFO:tasks.ceph.mon.c.smithi151.stderr: 8: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f26312f2c65] 2023-10-18T16:12:53.113 INFO:tasks.ceph.mon.c.smithi151.stderr: 9: /usr/lib64/ceph/libceph-common.so.2(+0x279e2e) [0x7f26312f2e2e] 2023-10-18T16:12:53.113 INFO:tasks.ceph.mon.c.smithi151.stderr: 10: (Monitor::~Monitor()+0xe40) [0x55edf4386b80] 2023-10-18T16:12:53.113 INFO:tasks.ceph.mon.c.smithi151.stderr: 11: (Monitor::~Monitor()+0xd) [0x55edf4386bdd] 2023-10-18T16:12:53.113 INFO:tasks.ceph.mon.c.smithi151.stderr: 12: main() 2023-10-18T16:12:53.114 INFO:tasks.ceph.mon.c.smithi151.stderr: 13: __libc_start_main() 2023-10-18T16:12:53.114 INFO:tasks.ceph.mon.c.smithi151.stderr: 14: _start() 2023-10-18T16:12:53.584 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T16:12:53.653 INFO:tasks.ceph.mon.c.smithi151.stderr:daemon-helper: command crashed with signal 11 2023-10-18T16:12:53.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:53.705+0000 7f3e54688700 1 Processor -- start 2023-10-18T16:12:53.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:53.705+0000 7f3e54688700 1 -- start start 2023-10-18T16:12:53.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:53.706+0000 7f3e54688700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e4c0c0610 0x7f3e4c0bff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:53.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:53.706+0000 7f3e54688700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e4c14dae0 0x7f3e4c14df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:53.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:53.706+0000 7f3e54688700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e4c0b6650 con 0x7f3e4c0bfb30 2023-10-18T16:12:53.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:53.706+0000 7f3e54688700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e4c14e460 con 0x7f3e4c0c0610 2023-10-18T16:12:53.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:53.706+0000 7f3e54688700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4c14d650 con 0x7f3e4c14dae0 2023-10-18T16:12:53.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:53.706+0000 7f3e52424700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e4c0bfb30 legacy=0x7f3e4c0bc790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:12:53.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:53.706+0000 7f3e52c25700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e4c14dae0 msgr2=0x7f3e4c14df70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:12:53.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:53.706+0000 7f3e51c23700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e4c0c0610 msgr2=0x7f3e4c0bff10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:12:53.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:53.706+0000 7f3e52c25700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e4c14dae0 0x7f3e4c14df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:12:53.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:53.706+0000 7f3e51c23700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e4c0c0610 0x7f3e4c0bff10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:12:53.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:53.908+0000 7f3e52424700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e4c0bfb30 legacy=0x7f3e4c0bc790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:12:53.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:53.908+0000 7f3e51c23700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e4c0c0610 msgr2=0x7f3e4c0bff10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:12:53.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:53.908+0000 7f3e52c25700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e4c14dae0 msgr2=0x7f3e4c14df70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:12:53.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:53.908+0000 7f3e51c23700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e4c0c0610 0x7f3e4c0bff10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:12:53.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:53.908+0000 7f3e52c25700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e4c14dae0 0x7f3e4c14df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:12:54.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:54.309+0000 7f3e52424700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e4c0bfb30 legacy=0x7f3e4c0bc790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:12:54.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:54.309+0000 7f3e52c25700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e4c14dae0 msgr2=0x7f3e4c14df70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:12:54.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:54.309+0000 7f3e51c23700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e4c0c0610 msgr2=0x7f3e4c0bff10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:12:54.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:54.309+0000 7f3e51c23700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e4c0c0610 0x7f3e4c0bff10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:12:54.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:54.309+0000 7f3e52c25700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e4c14dae0 0x7f3e4c14df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:12:55.112 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:55.109+0000 7f3e52c25700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e4c14dae0 msgr2=0x7f3e4c14df70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:12:55.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:55.109+0000 7f3e51c23700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e4c0c0610 msgr2=0x7f3e4c0bff10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:12:55.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:55.109+0000 7f3e52424700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e4c0bfb30 legacy=0x7f3e4c0bc790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:12:55.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:55.109+0000 7f3e52c25700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e4c14dae0 0x7f3e4c14df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:12:55.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:55.109+0000 7f3e51c23700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e4c0c0610 0x7f3e4c0bff10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:12:56.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:56.706+0000 7f3e43fff700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e4c14dae0 msgr2=0x7f3e4c14df70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:12:56.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:56.706+0000 7f3e43fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e4c14dae0 0x7f3e4c14df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:12:56.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:56.706+0000 7f3e43fff700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e4c0c0610 msgr2=0x7f3e4c0bff10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:12:56.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:56.706+0000 7f3e43fff700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e4c0c0610 0x7f3e4c0bff10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:12:56.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:56.706+0000 7f3e43fff700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e4c0bfb30 legacy=0x7f3e4c0bc790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:12:56.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:56.706+0000 7f3e43fff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30003590 0x7f3e30005a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:56.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:56.706+0000 7f3e43fff700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e4c05e0f0 con 0x7f3e30006010 2023-10-18T16:12:56.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:56.706+0000 7f3e43fff700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e4c14e460 con 0x7f3e30003590 2023-10-18T16:12:56.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:56.706+0000 7f3e51c23700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30006010 legacy=0x7f3e300084d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:12:56.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:56.706+0000 7f3e52424700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30003590 msgr2=0x7f3e30005a10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:12:56.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:56.706+0000 7f3e52424700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30003590 0x7f3e30005a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:12:56.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:56.907+0000 7f3e51c23700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30006010 legacy=0x7f3e300084d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:12:56.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:56.907+0000 7f3e52424700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30003590 msgr2=0x7f3e30005a10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:12:56.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:56.907+0000 7f3e52424700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30003590 0x7f3e30005a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:12:57.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:57.308+0000 7f3e51c23700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30006010 legacy=0x7f3e300084d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:12:57.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:57.308+0000 7f3e52424700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30003590 msgr2=0x7f3e30005a10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:12:57.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:57.308+0000 7f3e52424700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30003590 0x7f3e30005a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:12:58.112 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:58.109+0000 7f3e52424700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30003590 msgr2=0x7f3e30005a10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:12:58.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:58.109+0000 7f3e51c23700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30006010 legacy=0x7f3e300084d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:12:58.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:58.109+0000 7f3e52424700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30003590 0x7f3e30005a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:12:59.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:59.707+0000 7f3e43fff700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30003590 msgr2=0x7f3e30005a10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:12:59.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:59.707+0000 7f3e43fff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30003590 0x7f3e30005a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:12:59.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:59.707+0000 7f3e43fff700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30006010 legacy=0x7f3e300084d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:12:59.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:59.707+0000 7f3e52c25700 1 -- reap_dead start 2023-10-18T16:12:59.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:59.707+0000 7f3e43fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30008c00 0x7f3e30005a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:12:59.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:59.707+0000 7f3e43fff700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e4c0b6650 con 0x7f3e30009080 2023-10-18T16:12:59.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:59.707+0000 7f3e43fff700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e4c05e0f0 con 0x7f3e30009500 2023-10-18T16:12:59.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:59.707+0000 7f3e43fff700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4c14e460 con 0x7f3e30008c00 2023-10-18T16:12:59.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:59.707+0000 7f3e51c23700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30009080 legacy=0x7f3e300084d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:12:59.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:59.707+0000 7f3e52c25700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e30009500 legacy=0x7f3e3000f1d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:12:59.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:59.707+0000 7f3e52424700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30008c00 msgr2=0x7f3e30005a10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:12:59.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:59.707+0000 7f3e52424700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30008c00 0x7f3e30005a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:12:59.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:59.908+0000 7f3e51c23700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30009080 legacy=0x7f3e300084d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:12:59.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:59.908+0000 7f3e52424700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30008c00 msgr2=0x7f3e30005a10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:12:59.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:59.908+0000 7f3e52c25700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e30009500 legacy=0x7f3e3000f1d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:12:59.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:12:59.908+0000 7f3e52424700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30008c00 0x7f3e30005a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:13:00.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:00.309+0000 7f3e51c23700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30009080 legacy=0x7f3e300084d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:13:00.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:00.309+0000 7f3e52c25700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e30009500 legacy=0x7f3e3000f1d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:13:00.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:00.309+0000 7f3e52424700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30008c00 msgr2=0x7f3e30005a10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:13:00.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:00.309+0000 7f3e52424700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30008c00 0x7f3e30005a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:13:01.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:01.110+0000 7f3e51c23700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30009080 legacy=0x7f3e300084d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:13:01.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:01.110+0000 7f3e52424700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30008c00 msgr2=0x7f3e30005a10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:13:01.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:01.110+0000 7f3e52c25700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e30009500 legacy=0x7f3e3000f1d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:13:01.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:01.110+0000 7f3e52424700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30008c00 0x7f3e30005a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:13:02.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:02.707+0000 7f3e43fff700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30008c00 msgr2=0x7f3e30005a10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:13:02.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:02.707+0000 7f3e43fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30008c00 0x7f3e30005a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:13:02.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:02.707+0000 7f3e43fff700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e30009500 legacy=0x7f3e3000f1d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:13:02.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:02.707+0000 7f3e43fff700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30009080 legacy=0x7f3e300084d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:13:02.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:02.707+0000 7f3e43fff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e300119a0 0x7f3e30011db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:13:02.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:02.707+0000 7f3e43fff700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e30012430 0x7f3e300108f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:13:02.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:02.707+0000 7f3e43fff700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e4c14e460 con 0x7f3e300119a0 2023-10-18T16:13:02.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:02.707+0000 7f3e43fff700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e4c14d650 con 0x7f3e30012430 2023-10-18T16:13:02.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:02.707+0000 7f3e52424700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e300119a0 msgr2=0x7f3e30011db0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:13:02.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:02.707+0000 7f3e52424700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e300119a0 0x7f3e30011db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:13:02.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:02.707+0000 7f3e51c23700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e30012430 msgr2=0x7f3e300108f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:13:02.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:02.707+0000 7f3e51c23700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e30012430 0x7f3e300108f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:13:02.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:02.908+0000 7f3e52424700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e300119a0 msgr2=0x7f3e30011db0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:13:02.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:02.908+0000 7f3e52424700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e300119a0 0x7f3e30011db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:13:02.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:02.908+0000 7f3e51c23700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e30012430 msgr2=0x7f3e300108f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:13:02.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:02.908+0000 7f3e51c23700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e30012430 0x7f3e300108f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:13:03.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:03.309+0000 7f3e52424700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e300119a0 msgr2=0x7f3e30011db0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:13:03.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:03.309+0000 7f3e52424700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e300119a0 0x7f3e30011db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:13:03.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:03.309+0000 7f3e51c23700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e30012430 msgr2=0x7f3e300108f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:13:03.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:03.309+0000 7f3e51c23700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e30012430 0x7f3e300108f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:13:04.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:04.110+0000 7f3e52424700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e300119a0 msgr2=0x7f3e30011db0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:13:04.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:04.110+0000 7f3e52424700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e300119a0 0x7f3e30011db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:13:04.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:04.110+0000 7f3e51c23700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e30012430 msgr2=0x7f3e300108f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:13:04.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:04.110+0000 7f3e51c23700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e30012430 0x7f3e300108f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:13:05.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:05.707+0000 7f3e43fff700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e30012430 msgr2=0x7f3e300108f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:13:05.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:05.707+0000 7f3e43fff700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e30012430 0x7f3e300108f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:13:05.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:05.707+0000 7f3e43fff700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e300119a0 msgr2=0x7f3e30011db0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:13:05.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:05.707+0000 7f3e43fff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e300119a0 0x7f3e30011db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:13:05.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:05.707+0000 7f3e52c25700 1 -- reap_dead start 2023-10-18T16:13:05.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:05.707+0000 7f3e43fff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30008c00 0x7f3e30005980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:13:05.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:05.707+0000 7f3e43fff700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e30011db0 0x7f3e30010ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:13:05.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:05.707+0000 7f3e43fff700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e300129a0 con 0x7f3e30009080 2023-10-18T16:13:05.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:05.707+0000 7f3e43fff700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e4c14e460 con 0x7f3e30008c00 2023-10-18T16:13:05.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:05.707+0000 7f3e43fff700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e4c14d650 con 0x7f3e30011db0 2023-10-18T16:13:05.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:05.707+0000 7f3e51c23700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30009080 legacy=0x7f3e30008470 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:13:05.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:05.707+0000 7f3e52424700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30008c00 msgr2=0x7f3e30005980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:13:05.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:05.707+0000 7f3e52424700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30008c00 0x7f3e30005980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:13:05.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:05.708+0000 7f3e52c25700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e30011db0 msgr2=0x7f3e30010ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:13:05.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:05.708+0000 7f3e52c25700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e30011db0 0x7f3e30010ea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:13:05.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:05.908+0000 7f3e51c23700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30009080 legacy=0x7f3e30008470 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:13:05.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:05.908+0000 7f3e52424700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30008c00 msgr2=0x7f3e30005980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:13:05.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:05.908+0000 7f3e52424700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30008c00 0x7f3e30005980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:13:05.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:05.909+0000 7f3e52c25700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e30011db0 msgr2=0x7f3e30010ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:13:05.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:05.909+0000 7f3e52c25700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e30011db0 0x7f3e30010ea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:13:06.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:06.309+0000 7f3e51c23700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30009080 legacy=0x7f3e30008470 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:13:06.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:06.309+0000 7f3e52424700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30008c00 msgr2=0x7f3e30005980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:13:06.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:06.309+0000 7f3e52424700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30008c00 0x7f3e30005980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:13:06.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:06.310+0000 7f3e52c25700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e30011db0 msgr2=0x7f3e30010ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:13:06.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:06.310+0000 7f3e52c25700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e30011db0 0x7f3e30010ea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:13:07.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:07.110+0000 7f3e51c23700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30009080 legacy=0x7f3e30008470 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:13:07.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:07.110+0000 7f3e52424700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30008c00 msgr2=0x7f3e30005980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:13:07.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:07.110+0000 7f3e52424700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30008c00 0x7f3e30005980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:13:07.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:07.111+0000 7f3e52c25700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e30011db0 msgr2=0x7f3e30010ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:13:07.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:07.111+0000 7f3e52c25700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e30011db0 0x7f3e30010ea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:13:08.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:08.707+0000 7f3e43fff700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e30011db0 msgr2=0x7f3e30010ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:13:08.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:08.707+0000 7f3e43fff700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e30011db0 0x7f3e30010ea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:13:08.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:08.707+0000 7f3e43fff700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30008c00 msgr2=0x7f3e30005980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:13:08.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:08.707+0000 7f3e43fff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30008c00 0x7f3e30005980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:13:08.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:08.707+0000 7f3e43fff700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30009080 legacy=0x7f3e30008470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:13:08.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:08.708+0000 7f3e43fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30014be0 0x7f3e30016fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:13:08.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:08.708+0000 7f3e43fff700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e4c0b6650 con 0x7f3e3000f1d0 2023-10-18T16:13:08.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:08.708+0000 7f3e43fff700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4c14e460 con 0x7f3e30014be0 2023-10-18T16:13:08.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:08.708+0000 7f3e52424700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e3000f1d0 legacy=0x7f3e3000f5b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:13:08.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:08.708+0000 7f3e51c23700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30014be0 msgr2=0x7f3e30016fd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:13:08.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:08.708+0000 7f3e51c23700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30014be0 0x7f3e30016fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:13:08.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:08.909+0000 7f3e51c23700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30014be0 msgr2=0x7f3e30016fd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:13:08.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:08.909+0000 7f3e52424700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e3000f1d0 legacy=0x7f3e3000f5b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:13:08.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:08.909+0000 7f3e51c23700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30014be0 0x7f3e30016fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:13:08.940 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-18T16:13:08.942 DEBUG:teuthology.orchestra.run.smithi151:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-18T16:13:09.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:09.310+0000 7f3e51c23700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30014be0 msgr2=0x7f3e30016fd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:13:09.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:09.310+0000 7f3e52424700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e3000f1d0 legacy=0x7f3e3000f5b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:13:09.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:09.310+0000 7f3e51c23700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30014be0 0x7f3e30016fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:13:10.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:10.111+0000 7f3e52424700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e3000f1d0 legacy=0x7f3e3000f5b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:13:10.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:10.111+0000 7f3e51c23700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30014be0 msgr2=0x7f3e30016fd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:13:10.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:10.111+0000 7f3e51c23700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30014be0 0x7f3e30016fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:13:11.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:11.708+0000 7f3e43fff700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30014be0 msgr2=0x7f3e30016fd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:13:11.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:11.708+0000 7f3e43fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30014be0 0x7f3e30016fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:13:11.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:11.708+0000 7f3e43fff700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e3000f1d0 legacy=0x7f3e3000f5b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:13:11.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:11.708+0000 7f3e52c25700 1 -- reap_dead start 2023-10-18T16:13:11.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:11.708+0000 7f3e43fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30005980 0x7f3e30010da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:13:11.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:11.708+0000 7f3e43fff700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e300129a0 con 0x7f3e30011db0 2023-10-18T16:13:11.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:11.708+0000 7f3e43fff700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e4c0b6650 con 0x7f3e3000a440 2023-10-18T16:13:11.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:11.708+0000 7f3e43fff700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4c14e460 con 0x7f3e30005980 2023-10-18T16:13:11.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:11.708+0000 7f3e52424700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30011db0 legacy=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:13:11.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:11.708+0000 7f3e51c23700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30005980 msgr2=0x7f3e30010da0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:13:11.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:11.708+0000 7f3e52c25700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e3000a440 legacy=0x7f3e3000f1d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:13:11.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:11.708+0000 7f3e51c23700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30005980 0x7f3e30010da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:13:11.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:11.909+0000 7f3e52424700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30011db0 legacy=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:13:11.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:11.909+0000 7f3e51c23700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30005980 msgr2=0x7f3e30010da0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:13:11.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:11.909+0000 7f3e52c25700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e3000a440 legacy=0x7f3e3000f1d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:13:11.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:11.909+0000 7f3e51c23700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30005980 0x7f3e30010da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:13:12.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:12.310+0000 7f3e52424700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30011db0 legacy=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:13:12.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:12.310+0000 7f3e52c25700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e3000a440 legacy=0x7f3e3000f1d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:13:12.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:12.310+0000 7f3e51c23700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30005980 msgr2=0x7f3e30010da0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:13:12.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:12.310+0000 7f3e51c23700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30005980 0x7f3e30010da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:13:13.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:13.111+0000 7f3e52424700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30011db0 legacy=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:13:13.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:13.111+0000 7f3e52c25700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e3000a440 legacy=0x7f3e3000f1d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:13:13.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:13.111+0000 7f3e51c23700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30005980 msgr2=0x7f3e30010da0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:13:13.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:13.111+0000 7f3e51c23700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30005980 0x7f3e30010da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:13:14.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:14.708+0000 7f3e43fff700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30005980 msgr2=0x7f3e30010da0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:13:14.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:14.708+0000 7f3e43fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30005980 0x7f3e30010da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:13:14.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:14.708+0000 7f3e43fff700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e3000a440 legacy=0x7f3e3000f1d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:13:14.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:14.708+0000 7f3e43fff700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30011db0 legacy=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:13:14.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:14.708+0000 7f3e43fff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e300108f0 0x7f3e3000a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:13:14.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:14.708+0000 7f3e43fff700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e30014b60 0x7f3e30016f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:13:14.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:14.708+0000 7f3e43fff700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e4c14e460 con 0x7f3e300108f0 2023-10-18T16:13:14.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:14.708+0000 7f3e43fff700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e4c14d650 con 0x7f3e30014b60 2023-10-18T16:13:14.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:14.708+0000 7f3e52424700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e300108f0 msgr2=0x7f3e3000a850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:13:14.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:14.708+0000 7f3e52424700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e300108f0 0x7f3e3000a850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:13:14.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:14.708+0000 7f3e51c23700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e30014b60 msgr2=0x7f3e30016f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:13:14.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:14.708+0000 7f3e51c23700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e30014b60 0x7f3e30016f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:13:14.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:14.909+0000 7f3e52424700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e300108f0 msgr2=0x7f3e3000a850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:13:14.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:14.909+0000 7f3e52424700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e300108f0 0x7f3e3000a850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:13:14.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:14.909+0000 7f3e51c23700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e30014b60 msgr2=0x7f3e30016f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:13:14.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:14.909+0000 7f3e51c23700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e30014b60 0x7f3e30016f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:13:15.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:15.310+0000 7f3e52424700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e300108f0 msgr2=0x7f3e3000a850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:13:15.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:15.310+0000 7f3e52424700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e300108f0 0x7f3e3000a850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:13:15.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:15.310+0000 7f3e51c23700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e30014b60 msgr2=0x7f3e30016f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:13:15.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:15.310+0000 7f3e51c23700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e30014b60 0x7f3e30016f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:13:16.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:16.111+0000 7f3e52424700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e300108f0 msgr2=0x7f3e3000a850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:13:16.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:16.111+0000 7f3e52424700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e300108f0 0x7f3e3000a850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:13:16.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:16.111+0000 7f3e51c23700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e30014b60 msgr2=0x7f3e30016f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:13:16.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:16.111+0000 7f3e51c23700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e30014b60 0x7f3e30016f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:13:17.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:17.709+0000 7f3e43fff700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e30014b60 msgr2=0x7f3e30016f50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:13:17.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:17.709+0000 7f3e43fff700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e30014b60 0x7f3e30016f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:13:17.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:17.709+0000 7f3e43fff700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e300108f0 msgr2=0x7f3e3000a850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:13:17.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:17.709+0000 7f3e43fff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e300108f0 0x7f3e3000a850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:13:17.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:17.709+0000 7f3e52c25700 1 -- reap_dead start 2023-10-18T16:13:17.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:17.709+0000 7f3e43fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30005980 0x7f3e3000a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:13:17.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:17.709+0000 7f3e43fff700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e4c05e0f0 con 0x7f3e30011db0 2023-10-18T16:13:17.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:17.709+0000 7f3e52424700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30011db0 legacy=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:13:17.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:17.709+0000 7f3e43fff700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e300129a0 con 0x7f3e3000a440 2023-10-18T16:13:17.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:17.709+0000 7f3e43fff700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4c14e460 con 0x7f3e30005980 2023-10-18T16:13:17.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:17.709+0000 7f3e51c23700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e3000a440 legacy=0x7f3e3000f1d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:13:17.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:17.709+0000 7f3e52c25700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30005980 msgr2=0x7f3e3000a850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:13:17.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:17.709+0000 7f3e52c25700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30005980 0x7f3e3000a850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:13:17.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:17.910+0000 7f3e52424700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30011db0 legacy=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:13:17.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:17.910+0000 7f3e51c23700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e3000a440 legacy=0x7f3e3000f1d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:13:17.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:17.910+0000 7f3e52c25700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30005980 msgr2=0x7f3e3000a850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:13:17.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:17.910+0000 7f3e52c25700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30005980 0x7f3e3000a850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:13:18.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:18.311+0000 7f3e52424700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30011db0 legacy=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:13:18.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:18.311+0000 7f3e51c23700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e3000a440 legacy=0x7f3e3000f1d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:13:18.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:18.311+0000 7f3e52c25700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30005980 msgr2=0x7f3e3000a850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:13:18.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:18.311+0000 7f3e52c25700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30005980 0x7f3e3000a850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:13:19.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:19.112+0000 7f3e52424700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30011db0 legacy=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:13:19.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:19.112+0000 7f3e52c25700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30005980 msgr2=0x7f3e3000a850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:13:19.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:19.112+0000 7f3e51c23700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e3000a440 legacy=0x7f3e3000f1d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:13:19.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:19.112+0000 7f3e52c25700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30005980 0x7f3e3000a850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:13:20.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:20.709+0000 7f3e43fff700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30005980 msgr2=0x7f3e3000a850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:13:20.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:20.709+0000 7f3e43fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30005980 0x7f3e3000a850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:13:20.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:20.709+0000 7f3e43fff700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e3000a440 legacy=0x7f3e3000f1d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:13:20.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:20.709+0000 7f3e43fff700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30011db0 legacy=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:13:20.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:20.709+0000 7f3e43fff700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e300108f0 0x7f3e30010d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:13:20.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:20.709+0000 7f3e43fff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30014b40 0x7f3e30016f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:13:20.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:20.709+0000 7f3e43fff700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e4c14e460 con 0x7f3e30014b40 2023-10-18T16:13:20.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:20.709+0000 7f3e43fff700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e4c14d650 con 0x7f3e300108f0 2023-10-18T16:13:20.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:20.709+0000 7f3e51c23700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30014b40 msgr2=0x7f3e30016f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:13:20.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:20.709+0000 7f3e51c23700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30014b40 0x7f3e30016f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:13:20.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:20.709+0000 7f3e52424700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e300108f0 msgr2=0x7f3e30010d80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:13:20.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:20.709+0000 7f3e52424700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e300108f0 0x7f3e30010d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:13:20.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:20.910+0000 7f3e51c23700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30014b40 msgr2=0x7f3e30016f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:13:20.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:20.910+0000 7f3e51c23700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30014b40 0x7f3e30016f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:13:20.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:20.910+0000 7f3e52424700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e300108f0 msgr2=0x7f3e30010d80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:13:20.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:20.910+0000 7f3e52424700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e300108f0 0x7f3e30010d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:13:21.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:21.311+0000 7f3e51c23700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30014b40 msgr2=0x7f3e30016f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:13:21.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:21.311+0000 7f3e51c23700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30014b40 0x7f3e30016f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:13:21.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:21.311+0000 7f3e52424700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e300108f0 msgr2=0x7f3e30010d80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:13:21.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:21.311+0000 7f3e52424700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e300108f0 0x7f3e30010d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:13:22.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:22.112+0000 7f3e51c23700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30014b40 msgr2=0x7f3e30016f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:13:22.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:22.112+0000 7f3e51c23700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30014b40 0x7f3e30016f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:13:22.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:22.112+0000 7f3e52424700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e300108f0 msgr2=0x7f3e30010d80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:13:22.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:22.112+0000 7f3e52424700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e300108f0 0x7f3e30010d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:13:23.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:23.709+0000 7f3e43fff700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e300108f0 msgr2=0x7f3e30010d80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:13:23.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:23.709+0000 7f3e43fff700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e300108f0 0x7f3e30010d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:13:23.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:23.709+0000 7f3e43fff700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30014b40 msgr2=0x7f3e30016f60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:13:23.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:23.709+0000 7f3e43fff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30014b40 0x7f3e30016f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:13:23.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:23.709+0000 7f3e52c25700 1 -- reap_dead start 2023-10-18T16:13:23.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:23.709+0000 7f3e43fff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30010d80 0x7f3e300101e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:13:23.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:23.709+0000 7f3e43fff700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e4c0b6650 con 0x7f3e30011db0 2023-10-18T16:13:23.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:23.709+0000 7f3e43fff700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e4c05e0f0 con 0x7f3e30005980 2023-10-18T16:13:23.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:23.709+0000 7f3e43fff700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e4c14e460 con 0x7f3e30010d80 2023-10-18T16:13:23.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:23.709+0000 7f3e52424700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30011db0 legacy=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:13:23.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:23.709+0000 7f3e52c25700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30010d80 msgr2=0x7f3e300101e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:13:23.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:23.709+0000 7f3e52c25700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30010d80 0x7f3e300101e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:13:23.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:23.709+0000 7f3e51c23700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e30005980 legacy=0x7f3e3000a440 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:13:23.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:23.910+0000 7f3e52424700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30011db0 legacy=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:13:23.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:23.910+0000 7f3e52c25700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30010d80 msgr2=0x7f3e300101e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:13:23.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:23.910+0000 7f3e52c25700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30010d80 0x7f3e300101e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:13:23.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:23.910+0000 7f3e51c23700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e30005980 legacy=0x7f3e3000a440 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:13:24.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:24.311+0000 7f3e52c25700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30010d80 msgr2=0x7f3e300101e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:13:24.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:24.311+0000 7f3e52424700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30011db0 legacy=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:13:24.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:24.311+0000 7f3e52c25700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30010d80 0x7f3e300101e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:13:24.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:24.311+0000 7f3e51c23700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e30005980 legacy=0x7f3e3000a440 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:13:25.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:25.112+0000 7f3e52424700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30011db0 legacy=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:13:25.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:25.112+0000 7f3e52c25700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30010d80 msgr2=0x7f3e300101e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:13:25.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:25.112+0000 7f3e52c25700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30010d80 0x7f3e300101e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:13:25.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:25.112+0000 7f3e51c23700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e30005980 legacy=0x7f3e3000a440 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:13:26.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:26.710+0000 7f3e43fff700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30010d80 msgr2=0x7f3e300101e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:13:26.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:26.710+0000 7f3e43fff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30010d80 0x7f3e300101e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:13:26.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:26.710+0000 7f3e43fff700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e30005980 legacy=0x7f3e3000a440 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:13:26.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:26.710+0000 7f3e43fff700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30011db0 legacy=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:13:26.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:26.710+0000 7f3e43fff700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e300106d0 0x7f3e30014b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:13:26.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:26.710+0000 7f3e43fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30015030 0x7f3e30017420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:13:26.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:26.710+0000 7f3e43fff700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e4c14e460 con 0x7f3e300106d0 2023-10-18T16:13:26.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:26.710+0000 7f3e43fff700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4c14d650 con 0x7f3e30015030 2023-10-18T16:13:26.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:26.710+0000 7f3e51c23700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30015030 msgr2=0x7f3e30017420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:13:26.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:26.710+0000 7f3e52424700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e300106d0 msgr2=0x7f3e30014b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:13:26.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:26.710+0000 7f3e51c23700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30015030 0x7f3e30017420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:13:26.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:26.710+0000 7f3e52424700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e300106d0 0x7f3e30014b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:13:26.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:26.911+0000 7f3e51c23700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30015030 msgr2=0x7f3e30017420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:13:26.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:26.911+0000 7f3e51c23700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30015030 0x7f3e30017420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:13:26.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:26.911+0000 7f3e52424700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e300106d0 msgr2=0x7f3e30014b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:13:26.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:26.911+0000 7f3e52424700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e300106d0 0x7f3e30014b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:13:27.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:27.312+0000 7f3e51c23700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30015030 msgr2=0x7f3e30017420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:13:27.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:27.312+0000 7f3e52424700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e300106d0 msgr2=0x7f3e30014b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:13:27.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:27.312+0000 7f3e51c23700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30015030 0x7f3e30017420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:13:27.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:27.312+0000 7f3e52424700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e300106d0 0x7f3e30014b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:13:28.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:28.113+0000 7f3e51c23700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30015030 msgr2=0x7f3e30017420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:13:28.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:28.113+0000 7f3e51c23700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30015030 0x7f3e30017420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:13:28.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:28.113+0000 7f3e52424700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e300106d0 msgr2=0x7f3e30014b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:13:28.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:28.113+0000 7f3e52424700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e300106d0 0x7f3e30014b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:13:29.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:29.710+0000 7f3e43fff700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30015030 msgr2=0x7f3e30017420 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:13:29.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:29.710+0000 7f3e43fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30015030 0x7f3e30017420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:13:29.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:29.710+0000 7f3e43fff700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e300106d0 msgr2=0x7f3e30014b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:13:29.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:29.710+0000 7f3e43fff700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e300106d0 0x7f3e30014b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:13:29.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:29.710+0000 7f3e52c25700 1 -- reap_dead start 2023-10-18T16:13:29.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:29.710+0000 7f3e43fff700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e30017910 0x7f3e30019d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:13:29.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:29.710+0000 7f3e43fff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30010d80 0x7f3e30009d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:13:29.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:29.710+0000 7f3e43fff700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e300129a0 con 0x7f3e30011db0 2023-10-18T16:13:29.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:29.710+0000 7f3e43fff700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e4c14e460 con 0x7f3e30010d80 2023-10-18T16:13:29.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:29.710+0000 7f3e43fff700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e4c14d650 con 0x7f3e30017910 2023-10-18T16:13:29.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:29.710+0000 7f3e51c23700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30010d80 msgr2=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:13:29.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:29.710+0000 7f3e51c23700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30010d80 0x7f3e30009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:13:29.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:29.710+0000 7f3e52c25700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e30011db0 legacy=0x7f3e3000a220 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:13:29.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:29.710+0000 7f3e52424700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e30017910 msgr2=0x7f3e30019d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:13:29.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:29.710+0000 7f3e52424700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e30017910 0x7f3e30019d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:13:29.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:29.911+0000 7f3e51c23700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30010d80 msgr2=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:13:29.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:29.911+0000 7f3e51c23700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30010d80 0x7f3e30009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:13:29.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:29.911+0000 7f3e52c25700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e30011db0 legacy=0x7f3e3000a220 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:13:29.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:29.911+0000 7f3e52424700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e30017910 msgr2=0x7f3e30019d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:13:29.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:29.911+0000 7f3e52424700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e30017910 0x7f3e30019d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:13:30.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:30.312+0000 7f3e51c23700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30010d80 msgr2=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:13:30.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:30.312+0000 7f3e51c23700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30010d80 0x7f3e30009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:13:30.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:30.312+0000 7f3e52424700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e30017910 msgr2=0x7f3e30019d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:13:30.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:30.312+0000 7f3e52c25700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e30011db0 legacy=0x7f3e3000a220 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:13:30.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:30.312+0000 7f3e52424700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e30017910 0x7f3e30019d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:13:31.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:31.113+0000 7f3e51c23700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30010d80 msgr2=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:13:31.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:31.113+0000 7f3e51c23700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30010d80 0x7f3e30009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:13:31.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:31.113+0000 7f3e52c25700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e30011db0 legacy=0x7f3e3000a220 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:13:31.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:31.113+0000 7f3e52424700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e30017910 msgr2=0x7f3e30019d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:13:31.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:31.113+0000 7f3e52424700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e30017910 0x7f3e30019d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:13:32.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:32.710+0000 7f3e43fff700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e30017910 msgr2=0x7f3e30019d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:13:32.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:32.710+0000 7f3e43fff700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e30017910 0x7f3e30019d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:13:32.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:32.710+0000 7f3e43fff700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30010d80 msgr2=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:13:32.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:32.710+0000 7f3e43fff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30010d80 0x7f3e30009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:13:32.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:32.710+0000 7f3e43fff700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e30011db0 legacy=0x7f3e3000a220 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:13:32.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:32.710+0000 7f3e43fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30005980 0x7f3e3000f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:13:32.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:32.710+0000 7f3e43fff700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e4c0b6650 con 0x7f3e3000a930 2023-10-18T16:13:32.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:32.710+0000 7f3e43fff700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4c14e460 con 0x7f3e30005980 2023-10-18T16:13:32.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:32.710+0000 7f3e51c23700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e3000a930 legacy=0x7f3e3000f6c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:13:32.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:32.710+0000 7f3e52424700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30005980 msgr2=0x7f3e3000f1d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:13:32.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:32.710+0000 7f3e52424700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30005980 0x7f3e3000f1d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:13:32.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:32.911+0000 7f3e51c23700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e3000a930 legacy=0x7f3e3000f6c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:13:32.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:32.911+0000 7f3e52424700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30005980 msgr2=0x7f3e3000f1d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:13:32.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:32.911+0000 7f3e52424700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30005980 0x7f3e3000f1d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:13:33.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:33.312+0000 7f3e51c23700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e3000a930 legacy=0x7f3e3000f6c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:13:33.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:33.312+0000 7f3e52424700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30005980 msgr2=0x7f3e3000f1d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:13:33.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:33.312+0000 7f3e52424700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30005980 0x7f3e3000f1d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:13:34.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:34.113+0000 7f3e51c23700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e3000a930 legacy=0x7f3e3000f6c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:13:34.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:34.113+0000 7f3e52424700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30005980 msgr2=0x7f3e3000f1d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:13:34.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:34.113+0000 7f3e52424700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30005980 0x7f3e3000f1d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:13:35.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:35.711+0000 7f3e43fff700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30005980 msgr2=0x7f3e3000f1d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:13:35.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:35.711+0000 7f3e43fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30005980 0x7f3e3000f1d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:13:35.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:35.711+0000 7f3e43fff700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e3000a930 legacy=0x7f3e3000f6c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:13:35.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:35.711+0000 7f3e52c25700 1 -- reap_dead start 2023-10-18T16:13:35.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:35.711+0000 7f3e43fff700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e30010d80 0x7f3e30009d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:13:35.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:35.711+0000 7f3e43fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30011db0 0x7f3e3000a220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:13:35.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:35.711+0000 7f3e43fff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30005980 0x7f3e3000a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:13:35.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:35.711+0000 7f3e43fff700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e4c14e460 con 0x7f3e30005980 2023-10-18T16:13:35.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:35.711+0000 7f3e43fff700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e4c14d650 con 0x7f3e30010d80 2023-10-18T16:13:35.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:35.711+0000 7f3e43fff700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e3000ac00 con 0x7f3e30011db0 2023-10-18T16:13:35.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:35.711+0000 7f3e52c25700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30005980 msgr2=0x7f3e3000a710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:13:35.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:35.711+0000 7f3e52c25700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30005980 0x7f3e3000a710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:13:35.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:35.711+0000 7f3e51c23700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30011db0 msgr2=0x7f3e3000a220 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:13:35.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:35.711+0000 7f3e52424700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e30010d80 msgr2=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:13:35.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:35.711+0000 7f3e51c23700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30011db0 0x7f3e3000a220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:13:35.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:35.711+0000 7f3e52424700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e30010d80 0x7f3e30009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:13:35.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:35.912+0000 7f3e52c25700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30005980 msgr2=0x7f3e3000a710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:13:35.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:35.912+0000 7f3e52c25700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30005980 0x7f3e3000a710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:13:35.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:35.912+0000 7f3e52424700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e30010d80 msgr2=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:13:35.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:35.912+0000 7f3e51c23700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30011db0 msgr2=0x7f3e3000a220 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:13:35.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:35.912+0000 7f3e52424700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e30010d80 0x7f3e30009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:13:35.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:35.912+0000 7f3e51c23700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30011db0 0x7f3e3000a220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:13:36.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:36.313+0000 7f3e52c25700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30005980 msgr2=0x7f3e3000a710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:13:36.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:36.313+0000 7f3e52c25700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30005980 0x7f3e3000a710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:13:36.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:36.313+0000 7f3e51c23700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30011db0 msgr2=0x7f3e3000a220 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:13:36.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:36.313+0000 7f3e52424700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e30010d80 msgr2=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:13:36.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:36.313+0000 7f3e51c23700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30011db0 0x7f3e3000a220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:13:36.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:36.313+0000 7f3e52424700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e30010d80 0x7f3e30009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:13:37.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:37.114+0000 7f3e52c25700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30005980 msgr2=0x7f3e3000a710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:13:37.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:37.114+0000 7f3e52c25700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30005980 0x7f3e3000a710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:13:37.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:37.114+0000 7f3e51c23700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30011db0 msgr2=0x7f3e3000a220 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:13:37.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:37.114+0000 7f3e52424700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e30010d80 msgr2=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:13:37.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:37.114+0000 7f3e51c23700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30011db0 0x7f3e3000a220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:13:37.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:37.114+0000 7f3e52424700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e30010d80 0x7f3e30009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:13:38.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:38.711+0000 7f3e43fff700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30011db0 msgr2=0x7f3e3000a220 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:13:38.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:38.711+0000 7f3e43fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30011db0 0x7f3e3000a220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:13:38.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:38.711+0000 7f3e43fff700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e30010d80 msgr2=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:13:38.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:38.711+0000 7f3e43fff700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e30010d80 0x7f3e30009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:13:38.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:38.711+0000 7f3e43fff700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30005980 msgr2=0x7f3e3000a710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:13:38.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:38.711+0000 7f3e43fff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30005980 0x7f3e3000a710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:13:38.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:38.711+0000 7f3e43fff700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e300129a0 con 0x7f3e30008470 2023-10-18T16:13:38.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:38.711+0000 7f3e43fff700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e4c0b6650 con 0x7f3e30008fd0 2023-10-18T16:13:38.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:38.711+0000 7f3e52424700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30008470 legacy=0x7f3e300088c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:13:38.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:38.711+0000 7f3e51c23700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e30008fd0 legacy=0x7f3e30016b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:13:38.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:38.912+0000 7f3e52424700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30008470 legacy=0x7f3e300088c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:13:38.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:38.912+0000 7f3e51c23700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e30008fd0 legacy=0x7f3e30016b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:13:39.022 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-18T16:13:39.025 DEBUG:teuthology.orchestra.run.smithi151:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-18T16:13:39.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:39.313+0000 7f3e52424700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30008470 legacy=0x7f3e300088c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:13:39.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:39.313+0000 7f3e51c23700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e30008fd0 legacy=0x7f3e30016b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:13:40.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:40.114+0000 7f3e52424700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30008470 legacy=0x7f3e300088c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:13:40.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:40.114+0000 7f3e51c23700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e30008fd0 legacy=0x7f3e30016b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:13:41.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:41.711+0000 7f3e43fff700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e30008fd0 legacy=0x7f3e30016b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:13:41.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:41.711+0000 7f3e43fff700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30008470 legacy=0x7f3e300088c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:13:41.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:41.711+0000 7f3e52c25700 1 -- reap_dead start 2023-10-18T16:13:41.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:41.711+0000 7f3e43fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30010cf0 0x7f3e300088c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:13:41.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:41.711+0000 7f3e43fff700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e4c05e0f0 con 0x7f3e30011db0 2023-10-18T16:13:41.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:41.711+0000 7f3e43fff700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e300129a0 con 0x7f3e30008fd0 2023-10-18T16:13:41.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:41.711+0000 7f3e43fff700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4c14e460 con 0x7f3e30010cf0 2023-10-18T16:13:41.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:41.711+0000 7f3e51c23700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30011db0 legacy=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:13:41.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:41.711+0000 7f3e52c25700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30010cf0 msgr2=0x7f3e300088c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:13:41.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:41.711+0000 7f3e52424700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e30008fd0 legacy=0x7f3e30016b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:13:41.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:41.711+0000 7f3e52c25700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30010cf0 0x7f3e300088c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:13:41.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:41.912+0000 7f3e51c23700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30011db0 legacy=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:13:41.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:41.912+0000 7f3e52c25700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30010cf0 msgr2=0x7f3e300088c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:13:41.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:41.912+0000 7f3e52424700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e30008fd0 legacy=0x7f3e30016b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:13:41.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:41.912+0000 7f3e52c25700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30010cf0 0x7f3e300088c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:13:42.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:42.313+0000 7f3e51c23700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30011db0 legacy=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:13:42.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:42.313+0000 7f3e52c25700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30010cf0 msgr2=0x7f3e300088c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:13:42.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:42.313+0000 7f3e52424700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e30008fd0 legacy=0x7f3e30016b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:13:42.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:42.313+0000 7f3e52c25700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30010cf0 0x7f3e300088c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:13:43.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:43.113+0000 7f3e51c23700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30011db0 legacy=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:13:43.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:43.113+0000 7f3e52c25700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30010cf0 msgr2=0x7f3e300088c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:13:43.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:43.113+0000 7f3e52424700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e30008fd0 legacy=0x7f3e30016b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:13:43.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:43.113+0000 7f3e52c25700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30010cf0 0x7f3e300088c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:13:44.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:44.711+0000 7f3e43fff700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30010cf0 msgr2=0x7f3e300088c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:13:44.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:44.711+0000 7f3e43fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30010cf0 0x7f3e300088c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:13:44.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:44.711+0000 7f3e43fff700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e30008fd0 legacy=0x7f3e30016b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:13:44.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:44.711+0000 7f3e43fff700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30011db0 legacy=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:13:44.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:44.711+0000 7f3e43fff700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e3000a440 0x7f3e30007450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:13:44.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:44.711+0000 7f3e43fff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30005980 0x7f3e30007940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:13:44.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:44.711+0000 7f3e43fff700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e4c14e460 con 0x7f3e30005980 2023-10-18T16:13:44.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:44.711+0000 7f3e43fff700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e4c14d650 con 0x7f3e3000a440 2023-10-18T16:13:44.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:44.711+0000 7f3e51c23700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30005980 msgr2=0x7f3e30007940 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:13:44.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:44.711+0000 7f3e51c23700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30005980 0x7f3e30007940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:13:44.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:44.711+0000 7f3e52424700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e3000a440 msgr2=0x7f3e30007450 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:13:44.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:44.711+0000 7f3e52424700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e3000a440 0x7f3e30007450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:13:44.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:44.912+0000 7f3e51c23700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30005980 msgr2=0x7f3e30007940 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:13:44.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:44.912+0000 7f3e51c23700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30005980 0x7f3e30007940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:13:44.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:44.912+0000 7f3e52424700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e3000a440 msgr2=0x7f3e30007450 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:13:44.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:44.912+0000 7f3e52424700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e3000a440 0x7f3e30007450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:13:45.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:45.313+0000 7f3e51c23700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30005980 msgr2=0x7f3e30007940 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:13:45.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:45.313+0000 7f3e51c23700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30005980 0x7f3e30007940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:13:45.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:45.313+0000 7f3e52424700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e3000a440 msgr2=0x7f3e30007450 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:13:45.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:45.313+0000 7f3e52424700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e3000a440 0x7f3e30007450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:13:46.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:46.114+0000 7f3e51c23700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30005980 msgr2=0x7f3e30007940 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:13:46.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:46.114+0000 7f3e51c23700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30005980 0x7f3e30007940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:13:46.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:46.114+0000 7f3e52424700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e3000a440 msgr2=0x7f3e30007450 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:13:46.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:46.114+0000 7f3e52424700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e3000a440 0x7f3e30007450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:13:47.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:47.711+0000 7f3e43fff700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e3000a440 msgr2=0x7f3e30007450 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:13:47.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:47.711+0000 7f3e43fff700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e3000a440 0x7f3e30007450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:13:47.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:47.711+0000 7f3e43fff700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30005980 msgr2=0x7f3e30007940 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:13:47.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:47.711+0000 7f3e43fff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30005980 0x7f3e30007940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:13:47.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:47.711+0000 7f3e52c25700 1 -- reap_dead start 2023-10-18T16:13:47.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:47.711+0000 7f3e43fff700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e30011db0 0x7f3e300088c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:13:47.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:47.711+0000 7f3e43fff700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e4c0b6650 con 0x7f3e3000a820 2023-10-18T16:13:47.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:47.711+0000 7f3e43fff700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e4c05e0f0 con 0x7f3e30010cf0 2023-10-18T16:13:47.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:47.711+0000 7f3e43fff700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e4c14e460 con 0x7f3e30011db0 2023-10-18T16:13:47.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:47.711+0000 7f3e52c25700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e3000a820 legacy=0x7f3e30004980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:13:47.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:47.711+0000 7f3e51c23700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e30011db0 msgr2=0x7f3e300088c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:13:47.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:47.711+0000 7f3e52424700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e30010cf0 legacy=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:13:47.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:47.711+0000 7f3e51c23700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e30011db0 0x7f3e300088c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:13:47.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:47.912+0000 7f3e52c25700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e3000a820 legacy=0x7f3e30004980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:13:47.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:47.912+0000 7f3e51c23700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e30011db0 msgr2=0x7f3e300088c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:13:47.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:47.912+0000 7f3e52424700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e30010cf0 legacy=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:13:47.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:47.912+0000 7f3e51c23700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e30011db0 0x7f3e300088c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:13:48.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:48.313+0000 7f3e52c25700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e3000a820 legacy=0x7f3e30004980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:13:48.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:48.313+0000 7f3e51c23700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e30011db0 msgr2=0x7f3e300088c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:13:48.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:48.313+0000 7f3e52424700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e30010cf0 legacy=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:13:48.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:48.313+0000 7f3e51c23700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e30011db0 0x7f3e300088c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:13:49.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:49.114+0000 7f3e52c25700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e3000a820 legacy=0x7f3e30004980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:13:49.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:49.114+0000 7f3e52424700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e30010cf0 legacy=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:13:49.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:49.114+0000 7f3e51c23700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e30011db0 msgr2=0x7f3e300088c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:13:49.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:49.114+0000 7f3e51c23700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e30011db0 0x7f3e300088c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:13:50.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:50.711+0000 7f3e43fff700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e30011db0 msgr2=0x7f3e300088c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:13:50.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:50.711+0000 7f3e43fff700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e30011db0 0x7f3e300088c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:13:50.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:50.711+0000 7f3e43fff700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e30010cf0 legacy=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:13:50.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:50.711+0000 7f3e43fff700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e3000a820 legacy=0x7f3e30004980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:13:50.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:50.711+0000 7f3e43fff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e300101e0 0x7f3e30008e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:13:50.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:50.711+0000 7f3e43fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30005090 0x7f3e300054a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:13:50.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:50.711+0000 7f3e43fff700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e4c14e460 con 0x7f3e300101e0 2023-10-18T16:13:50.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:50.711+0000 7f3e43fff700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4c14d650 con 0x7f3e30005090 2023-10-18T16:13:50.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:50.712+0000 7f3e52424700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e300101e0 msgr2=0x7f3e30008e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:13:50.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:50.712+0000 7f3e52424700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e300101e0 0x7f3e30008e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:13:50.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:50.712+0000 7f3e51c23700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30005090 msgr2=0x7f3e300054a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:13:50.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:50.712+0000 7f3e51c23700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30005090 0x7f3e300054a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:13:50.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:50.913+0000 7f3e52424700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e300101e0 msgr2=0x7f3e30008e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:13:50.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:50.913+0000 7f3e52424700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e300101e0 0x7f3e30008e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:13:50.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:50.913+0000 7f3e51c23700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30005090 msgr2=0x7f3e300054a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:13:50.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:50.913+0000 7f3e51c23700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30005090 0x7f3e300054a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:13:51.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:51.314+0000 7f3e52424700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e300101e0 msgr2=0x7f3e30008e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:13:51.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:51.314+0000 7f3e52424700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e300101e0 0x7f3e30008e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:13:51.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:51.314+0000 7f3e51c23700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30005090 msgr2=0x7f3e300054a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:13:51.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:51.314+0000 7f3e51c23700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30005090 0x7f3e300054a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:13:52.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:52.115+0000 7f3e52424700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e300101e0 msgr2=0x7f3e30008e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:13:52.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:52.115+0000 7f3e52424700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e300101e0 0x7f3e30008e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:13:52.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:52.115+0000 7f3e51c23700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30005090 msgr2=0x7f3e300054a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:13:52.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:52.115+0000 7f3e51c23700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30005090 0x7f3e300054a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:13:53.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:53.712+0000 7f3e43fff700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30005090 msgr2=0x7f3e300054a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:13:53.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:53.712+0000 7f3e43fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30005090 0x7f3e300054a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:13:53.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:53.712+0000 7f3e43fff700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e300101e0 msgr2=0x7f3e30008e50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:13:53.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:53.712+0000 7f3e43fff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e300101e0 0x7f3e30008e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:13:53.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:53.712+0000 7f3e52c25700 1 -- reap_dead start 2023-10-18T16:13:53.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:53.712+0000 7f3e43fff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30011db0 0x7f3e30004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:13:53.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:53.712+0000 7f3e43fff700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e300129a0 con 0x7f3e3000a820 2023-10-18T16:13:53.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:53.712+0000 7f3e43fff700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e4c0b6650 con 0x7f3e30010cf0 2023-10-18T16:13:53.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:53.712+0000 7f3e43fff700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e4c14e460 con 0x7f3e30011db0 2023-10-18T16:13:53.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:53.712+0000 7f3e52c25700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30011db0 msgr2=0x7f3e30004980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:13:53.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:53.712+0000 7f3e51c23700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e3000a820 legacy=0x7f3e300088c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:13:53.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:53.712+0000 7f3e52c25700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30011db0 0x7f3e30004980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:13:53.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:53.712+0000 7f3e52424700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e30010cf0 legacy=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:13:53.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:53.913+0000 7f3e51c23700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e3000a820 legacy=0x7f3e300088c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:13:53.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:53.913+0000 7f3e52c25700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30011db0 msgr2=0x7f3e30004980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:13:53.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:53.913+0000 7f3e52c25700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30011db0 0x7f3e30004980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:13:53.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:53.913+0000 7f3e52424700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e30010cf0 legacy=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:13:54.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:54.314+0000 7f3e52c25700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30011db0 msgr2=0x7f3e30004980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:13:54.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:54.314+0000 7f3e51c23700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e3000a820 legacy=0x7f3e300088c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:13:54.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:54.314+0000 7f3e52c25700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30011db0 0x7f3e30004980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:13:54.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:54.314+0000 7f3e52424700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e30010cf0 legacy=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:13:55.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:55.115+0000 7f3e52c25700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30011db0 msgr2=0x7f3e30004980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:13:55.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:55.115+0000 7f3e51c23700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e3000a820 legacy=0x7f3e300088c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:13:55.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:55.115+0000 7f3e52c25700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30011db0 0x7f3e30004980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:13:55.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:55.115+0000 7f3e52424700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e30010cf0 legacy=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:13:56.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:56.712+0000 7f3e43fff700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30011db0 msgr2=0x7f3e30004980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:13:56.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:56.712+0000 7f3e43fff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30011db0 0x7f3e30004980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:13:56.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:56.712+0000 7f3e43fff700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e30010cf0 legacy=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:13:56.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:56.712+0000 7f3e43fff700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e3000a820 legacy=0x7f3e300088c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:13:56.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:56.712+0000 7f3e43fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e300101e0 0x7f3e30004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:13:56.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:56.712+0000 7f3e43fff700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e30005360 0x7f3e30005770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:13:56.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:56.712+0000 7f3e43fff700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e4c14e460 con 0x7f3e30005360 2023-10-18T16:13:56.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:56.712+0000 7f3e43fff700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4c14d650 con 0x7f3e300101e0 2023-10-18T16:13:56.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:56.712+0000 7f3e51c23700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e30005360 msgr2=0x7f3e30005770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:13:56.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:56.712+0000 7f3e52424700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e300101e0 msgr2=0x7f3e30004e70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:13:56.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:56.712+0000 7f3e51c23700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e30005360 0x7f3e30005770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:13:56.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:56.712+0000 7f3e52424700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e300101e0 0x7f3e30004e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:13:56.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:56.913+0000 7f3e52424700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e300101e0 msgr2=0x7f3e30004e70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:13:56.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:56.913+0000 7f3e51c23700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e30005360 msgr2=0x7f3e30005770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:13:56.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:56.913+0000 7f3e52424700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e300101e0 0x7f3e30004e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:13:56.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:56.913+0000 7f3e51c23700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e30005360 0x7f3e30005770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:13:57.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:57.314+0000 7f3e51c23700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e30005360 msgr2=0x7f3e30005770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:13:57.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:57.314+0000 7f3e52424700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e300101e0 msgr2=0x7f3e30004e70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:13:57.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:57.314+0000 7f3e51c23700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e30005360 0x7f3e30005770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:13:57.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:57.314+0000 7f3e52424700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e300101e0 0x7f3e30004e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:13:58.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:58.115+0000 7f3e52424700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e300101e0 msgr2=0x7f3e30004e70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:13:58.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:58.115+0000 7f3e51c23700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e30005360 msgr2=0x7f3e30005770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:13:58.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:58.115+0000 7f3e52424700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e300101e0 0x7f3e30004e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:13:58.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:58.115+0000 7f3e51c23700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e30005360 0x7f3e30005770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:13:59.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:59.712+0000 7f3e43fff700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e300101e0 msgr2=0x7f3e30004e70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:13:59.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:59.712+0000 7f3e43fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e300101e0 0x7f3e30004e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:13:59.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:59.712+0000 7f3e43fff700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e30005360 msgr2=0x7f3e30005770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:13:59.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:59.712+0000 7f3e43fff700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e30005360 0x7f3e30005770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:13:59.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:59.712+0000 7f3e52c25700 1 -- reap_dead start 2023-10-18T16:13:59.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:59.712+0000 7f3e43fff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30010cf0 0x7f3e300088a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:13:59.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:59.712+0000 7f3e43fff700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e4c05e0f0 con 0x7f3e30011db0 2023-10-18T16:13:59.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:59.712+0000 7f3e43fff700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e300129a0 con 0x7f3e3000a820 2023-10-18T16:13:59.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:59.712+0000 7f3e43fff700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e4c14e460 con 0x7f3e30010cf0 2023-10-18T16:13:59.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:59.712+0000 7f3e51c23700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30010cf0 msgr2=0x7f3e300088a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:13:59.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:59.712+0000 7f3e52c25700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30011db0 legacy=0x7f3e30004980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:13:59.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:59.712+0000 7f3e51c23700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30010cf0 0x7f3e300088a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:13:59.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:59.712+0000 7f3e52424700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e3000a820 legacy=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:13:59.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:59.913+0000 7f3e51c23700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30010cf0 msgr2=0x7f3e300088a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:13:59.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:59.913+0000 7f3e52c25700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30011db0 legacy=0x7f3e30004980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:13:59.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:59.913+0000 7f3e51c23700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30010cf0 0x7f3e300088a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:13:59.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:13:59.913+0000 7f3e52424700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e3000a820 legacy=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:14:00.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:00.314+0000 7f3e51c23700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30010cf0 msgr2=0x7f3e300088a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:14:00.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:00.314+0000 7f3e52c25700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30011db0 legacy=0x7f3e30004980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:14:00.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:00.314+0000 7f3e51c23700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30010cf0 0x7f3e300088a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:14:00.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:00.314+0000 7f3e52424700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e3000a820 legacy=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:14:01.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:01.115+0000 7f3e52c25700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30011db0 legacy=0x7f3e30004980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:14:01.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:01.115+0000 7f3e51c23700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30010cf0 msgr2=0x7f3e300088a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:14:01.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:01.115+0000 7f3e51c23700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30010cf0 0x7f3e300088a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:14:01.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:01.115+0000 7f3e52424700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e3000a820 legacy=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:14:02.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:02.712+0000 7f3e43fff700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30010cf0 msgr2=0x7f3e300088a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:14:02.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:02.712+0000 7f3e43fff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30010cf0 0x7f3e300088a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:14:02.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:02.712+0000 7f3e43fff700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e3000a820 legacy=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:14:02.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:02.712+0000 7f3e43fff700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30011db0 legacy=0x7f3e30004980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:14:02.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:02.712+0000 7f3e43fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e300101e0 0x7f3e30008d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:14:02.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:02.713+0000 7f3e43fff700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e30005090 0x7f3e300054a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:14:02.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:02.713+0000 7f3e43fff700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e4c14e460 con 0x7f3e30005090 2023-10-18T16:14:02.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:02.713+0000 7f3e43fff700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4c14d650 con 0x7f3e300101e0 2023-10-18T16:14:02.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:02.713+0000 7f3e51c23700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e30005090 msgr2=0x7f3e300054a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:14:02.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:02.713+0000 7f3e52424700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e300101e0 msgr2=0x7f3e30008d90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:14:02.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:02.713+0000 7f3e51c23700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e30005090 0x7f3e300054a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:14:02.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:02.713+0000 7f3e52424700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e300101e0 0x7f3e30008d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:14:02.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:02.914+0000 7f3e51c23700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e30005090 msgr2=0x7f3e300054a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:14:02.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:02.914+0000 7f3e52424700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e300101e0 msgr2=0x7f3e30008d90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:14:02.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:02.914+0000 7f3e51c23700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e30005090 0x7f3e300054a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:14:02.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:02.914+0000 7f3e52424700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e300101e0 0x7f3e30008d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:14:03.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:03.315+0000 7f3e51c23700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e30005090 msgr2=0x7f3e300054a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:14:03.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:03.315+0000 7f3e52424700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e300101e0 msgr2=0x7f3e30008d90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:14:03.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:03.315+0000 7f3e51c23700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e30005090 0x7f3e300054a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:14:03.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:03.315+0000 7f3e52424700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e300101e0 0x7f3e30008d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:14:04.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:04.116+0000 7f3e51c23700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e30005090 msgr2=0x7f3e300054a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:14:04.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:04.116+0000 7f3e52424700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e300101e0 msgr2=0x7f3e30008d90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:14:04.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:04.116+0000 7f3e51c23700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e30005090 0x7f3e300054a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:14:04.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:04.116+0000 7f3e52424700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e300101e0 0x7f3e30008d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:14:05.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:05.713+0000 7f3e43fff700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e300101e0 msgr2=0x7f3e30008d90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:14:05.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:05.713+0000 7f3e43fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e300101e0 0x7f3e30008d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:14:05.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:05.713+0000 7f3e43fff700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e30005090 msgr2=0x7f3e300054a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:14:05.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:05.713+0000 7f3e43fff700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e30005090 0x7f3e300054a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:14:05.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:05.713+0000 7f3e52c25700 1 -- reap_dead start 2023-10-18T16:14:05.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:05.713+0000 7f3e43fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e300101e0 0x7f3e30008d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:14:05.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:05.713+0000 7f3e43fff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30005090 0x7f3e300088a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:14:05.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:05.713+0000 7f3e43fff700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e4c0b6650 con 0x7f3e30011db0 2023-10-18T16:14:05.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:05.713+0000 7f3e43fff700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e4c14e460 con 0x7f3e30005090 2023-10-18T16:14:05.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:05.713+0000 7f3e43fff700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4c14d650 con 0x7f3e300101e0 2023-10-18T16:14:05.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:05.713+0000 7f3e51c23700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30011db0 legacy=0x7f3e30004980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:14:05.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:05.713+0000 7f3e52c25700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30005090 msgr2=0x7f3e300088a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:14:05.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:05.713+0000 7f3e52424700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e300101e0 msgr2=0x7f3e30008d90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:14:05.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:05.713+0000 7f3e52c25700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30005090 0x7f3e300088a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:14:05.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:05.713+0000 7f3e52424700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e300101e0 0x7f3e30008d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:14:05.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:05.914+0000 7f3e51c23700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30011db0 legacy=0x7f3e30004980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:14:05.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:05.914+0000 7f3e52c25700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30005090 msgr2=0x7f3e300088a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:14:05.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:05.914+0000 7f3e52c25700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30005090 0x7f3e300088a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:14:05.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:05.914+0000 7f3e52424700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e300101e0 msgr2=0x7f3e30008d90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:14:05.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:05.914+0000 7f3e52424700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e300101e0 0x7f3e30008d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:14:06.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:06.315+0000 7f3e51c23700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30011db0 legacy=0x7f3e30004980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:14:06.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:06.315+0000 7f3e52c25700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30005090 msgr2=0x7f3e300088a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:14:06.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:06.315+0000 7f3e52c25700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30005090 0x7f3e300088a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:14:06.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:06.315+0000 7f3e52424700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e300101e0 msgr2=0x7f3e30008d90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:14:06.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:06.315+0000 7f3e52424700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e300101e0 0x7f3e30008d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:14:07.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:07.116+0000 7f3e51c23700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30011db0 legacy=0x7f3e30004980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:14:07.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:07.116+0000 7f3e52c25700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30005090 msgr2=0x7f3e300088a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:14:07.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:07.116+0000 7f3e52c25700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30005090 0x7f3e300088a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:14:07.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:07.116+0000 7f3e52424700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e300101e0 msgr2=0x7f3e30008d90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:14:07.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:07.116+0000 7f3e52424700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e300101e0 0x7f3e30008d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:14:08.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:08.713+0000 7f3e43fff700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e300101e0 msgr2=0x7f3e30008d90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:14:08.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:08.713+0000 7f3e43fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e300101e0 0x7f3e30008d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:14:08.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:08.713+0000 7f3e43fff700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30005090 msgr2=0x7f3e300088a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:14:08.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:08.713+0000 7f3e43fff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30005090 0x7f3e300088a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:14:08.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:08.713+0000 7f3e43fff700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30011db0 legacy=0x7f3e30004980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:14:08.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:08.713+0000 7f3e43fff700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e3000a820 0x7f3e30009d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:14:08.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:08.713+0000 7f3e43fff700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e4c05e0f0 con 0x7f3e30010cf0 2023-10-18T16:14:08.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:08.713+0000 7f3e43fff700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e4c14e460 con 0x7f3e3000a820 2023-10-18T16:14:08.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:08.713+0000 7f3e52424700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e3000a820 msgr2=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:14:08.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:08.713+0000 7f3e51c23700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e30010cf0 legacy=0x7f3e300054a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:14:08.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:08.713+0000 7f3e52424700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e3000a820 0x7f3e30009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:14:08.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:08.914+0000 7f3e51c23700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e30010cf0 legacy=0x7f3e300054a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:14:08.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:08.914+0000 7f3e52424700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e3000a820 msgr2=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:14:08.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:08.914+0000 7f3e52424700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e3000a820 0x7f3e30009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:14:09.091 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-18T16:14:09.094 DEBUG:teuthology.orchestra.run.smithi151:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-18T16:14:09.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:09.315+0000 7f3e52424700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e3000a820 msgr2=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:14:09.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:09.315+0000 7f3e51c23700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e30010cf0 legacy=0x7f3e300054a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:14:09.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:09.315+0000 7f3e52424700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e3000a820 0x7f3e30009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:14:10.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:10.116+0000 7f3e52424700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e3000a820 msgr2=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:14:10.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:10.116+0000 7f3e52424700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e3000a820 0x7f3e30009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:14:10.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:10.116+0000 7f3e51c23700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e30010cf0 legacy=0x7f3e300054a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:14:11.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:11.714+0000 7f3e43fff700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e3000a820 msgr2=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:14:11.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:11.714+0000 7f3e43fff700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e3000a820 0x7f3e30009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:14:11.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:11.714+0000 7f3e43fff700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e30010cf0 legacy=0x7f3e300054a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:14:11.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:11.714+0000 7f3e52c25700 1 -- reap_dead start 2023-10-18T16:14:11.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:11.714+0000 7f3e43fff700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e3000a820 0x7f3e30009d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:14:11.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:11.714+0000 7f3e43fff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30011db0 0x7f3e300054a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:14:11.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:11.714+0000 7f3e43fff700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e4c0b6650 con 0x7f3e30010cf0 2023-10-18T16:14:11.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:11.714+0000 7f3e43fff700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e4c14e460 con 0x7f3e30011db0 2023-10-18T16:14:11.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:11.714+0000 7f3e43fff700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e4c14d650 con 0x7f3e3000a820 2023-10-18T16:14:11.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:11.714+0000 7f3e51c23700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30011db0 msgr2=0x7f3e300054a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:14:11.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:11.714+0000 7f3e52c25700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30010cf0 legacy=0x7f3e300088a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:14:11.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:11.714+0000 7f3e51c23700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30011db0 0x7f3e300054a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:14:11.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:11.714+0000 7f3e52424700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e3000a820 msgr2=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:14:11.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:11.714+0000 7f3e52424700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e3000a820 0x7f3e30009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:14:11.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:11.915+0000 7f3e51c23700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30011db0 msgr2=0x7f3e300054a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:14:11.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:11.915+0000 7f3e52c25700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30010cf0 legacy=0x7f3e300088a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:14:11.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:11.915+0000 7f3e51c23700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30011db0 0x7f3e300054a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:14:11.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:11.915+0000 7f3e52424700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e3000a820 msgr2=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:14:11.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:11.915+0000 7f3e52424700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e3000a820 0x7f3e30009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:14:12.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:12.316+0000 7f3e52c25700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30010cf0 legacy=0x7f3e300088a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:14:12.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:12.316+0000 7f3e51c23700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30011db0 msgr2=0x7f3e300054a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:14:12.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:12.316+0000 7f3e51c23700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30011db0 0x7f3e300054a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:14:12.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:12.316+0000 7f3e52424700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e3000a820 msgr2=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:14:12.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:12.316+0000 7f3e52424700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e3000a820 0x7f3e30009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:14:13.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:13.117+0000 7f3e52c25700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30010cf0 legacy=0x7f3e300088a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:14:13.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:13.117+0000 7f3e51c23700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30011db0 msgr2=0x7f3e300054a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:14:13.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:13.117+0000 7f3e51c23700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30011db0 0x7f3e300054a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:14:13.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:13.117+0000 7f3e52424700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e3000a820 msgr2=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:14:13.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:13.117+0000 7f3e52424700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e3000a820 0x7f3e30009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:14:14.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:14.714+0000 7f3e43fff700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e3000a820 msgr2=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:14:14.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:14.714+0000 7f3e43fff700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e3000a820 0x7f3e30009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:14:14.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:14.714+0000 7f3e43fff700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30011db0 msgr2=0x7f3e300054a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:14:14.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:14.714+0000 7f3e43fff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30011db0 0x7f3e300054a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:14:14.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:14.714+0000 7f3e43fff700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30010cf0 legacy=0x7f3e300088a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:14:14.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:14.714+0000 7f3e43fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30005090 0x7f3e30018b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:14:14.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:14.714+0000 7f3e43fff700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e4c05e0f0 con 0x7f3e300101e0 2023-10-18T16:14:14.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:14.714+0000 7f3e43fff700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4c14e460 con 0x7f3e30005090 2023-10-18T16:14:14.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:14.714+0000 7f3e51c23700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30005090 msgr2=0x7f3e30018b60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:14:14.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:14.714+0000 7f3e52424700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e300101e0 legacy=0x7f3e30003970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:14:14.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:14.714+0000 7f3e51c23700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30005090 0x7f3e30018b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:14:14.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:14.915+0000 7f3e52424700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e300101e0 legacy=0x7f3e30003970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:14:14.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:14.915+0000 7f3e51c23700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30005090 msgr2=0x7f3e30018b60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:14:14.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:14.915+0000 7f3e51c23700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30005090 0x7f3e30018b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:14:15.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:15.316+0000 7f3e51c23700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30005090 msgr2=0x7f3e30018b60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:14:15.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:15.316+0000 7f3e52424700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e300101e0 legacy=0x7f3e30003970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:14:15.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:15.316+0000 7f3e51c23700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30005090 0x7f3e30018b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:14:16.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:16.117+0000 7f3e52424700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e300101e0 legacy=0x7f3e30003970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:14:16.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:16.117+0000 7f3e51c23700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30005090 msgr2=0x7f3e30018b60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:14:16.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:16.117+0000 7f3e51c23700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30005090 0x7f3e30018b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:14:17.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:17.714+0000 7f3e43fff700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30005090 msgr2=0x7f3e30018b60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:14:17.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:17.714+0000 7f3e43fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30005090 0x7f3e30018b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:14:17.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:17.714+0000 7f3e43fff700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e300101e0 legacy=0x7f3e30003970 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:14:17.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:17.714+0000 7f3e52c25700 1 -- reap_dead start 2023-10-18T16:14:17.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:17.714+0000 7f3e43fff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30005090 0x7f3e30009d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:14:17.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:17.714+0000 7f3e43fff700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e4c0b6650 con 0x7f3e3000a820 2023-10-18T16:14:17.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:17.714+0000 7f3e43fff700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e4c05e0f0 con 0x7f3e300101e0 2023-10-18T16:14:17.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:17.714+0000 7f3e43fff700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e4c14e460 con 0x7f3e30005090 2023-10-18T16:14:17.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:17.714+0000 7f3e51c23700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30005090 msgr2=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:14:17.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:17.714+0000 7f3e52c25700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e3000a820 legacy=0x7f3e300054a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:14:17.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:17.714+0000 7f3e51c23700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30005090 0x7f3e30009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:14:17.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:17.714+0000 7f3e52424700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e300101e0 legacy=0x7f3e30003970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:14:17.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:17.915+0000 7f3e52c25700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e3000a820 legacy=0x7f3e300054a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:14:17.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:17.915+0000 7f3e51c23700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30005090 msgr2=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:14:17.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:17.915+0000 7f3e51c23700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30005090 0x7f3e30009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:14:17.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:17.915+0000 7f3e52424700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e300101e0 legacy=0x7f3e30003970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:14:18.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:18.316+0000 7f3e52c25700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e3000a820 legacy=0x7f3e300054a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:14:18.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:18.316+0000 7f3e51c23700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30005090 msgr2=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:14:18.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:18.316+0000 7f3e51c23700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30005090 0x7f3e30009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:14:18.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:18.316+0000 7f3e52424700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e300101e0 legacy=0x7f3e30003970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:14:19.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:19.117+0000 7f3e51c23700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30005090 msgr2=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:14:19.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:19.117+0000 7f3e52c25700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e3000a820 legacy=0x7f3e300054a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:14:19.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:19.117+0000 7f3e51c23700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30005090 0x7f3e30009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:14:19.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:19.117+0000 7f3e52424700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e300101e0 legacy=0x7f3e30003970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:14:20.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:20.715+0000 7f3e43fff700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30005090 msgr2=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:14:20.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:20.715+0000 7f3e43fff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30005090 0x7f3e30009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:14:20.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:20.715+0000 7f3e43fff700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e300101e0 legacy=0x7f3e30003970 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:14:20.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:20.715+0000 7f3e43fff700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e3000a820 legacy=0x7f3e300054a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:14:20.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:20.715+0000 7f3e43fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30010cf0 0x7f3e300088a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:14:20.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:20.715+0000 7f3e43fff700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e30011db0 0x7f3e30008d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:14:20.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:20.715+0000 7f3e43fff700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e4c14e460 con 0x7f3e30011db0 2023-10-18T16:14:20.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:20.715+0000 7f3e43fff700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4c14d650 con 0x7f3e30010cf0 2023-10-18T16:14:20.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:20.715+0000 7f3e52424700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30010cf0 msgr2=0x7f3e300088a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:14:20.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:20.715+0000 7f3e51c23700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e30011db0 msgr2=0x7f3e30008d90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:14:20.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:20.715+0000 7f3e52424700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30010cf0 0x7f3e300088a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:14:20.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:20.715+0000 7f3e51c23700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e30011db0 0x7f3e30008d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:14:20.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:20.916+0000 7f3e52424700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30010cf0 msgr2=0x7f3e300088a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:14:20.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:20.916+0000 7f3e51c23700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e30011db0 msgr2=0x7f3e30008d90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:14:20.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:20.916+0000 7f3e52424700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30010cf0 0x7f3e300088a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:14:20.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:20.916+0000 7f3e51c23700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e30011db0 0x7f3e30008d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:14:21.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:21.317+0000 7f3e52424700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30010cf0 msgr2=0x7f3e300088a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:14:21.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:21.317+0000 7f3e51c23700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e30011db0 msgr2=0x7f3e30008d90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:14:21.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:21.317+0000 7f3e52424700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30010cf0 0x7f3e300088a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:14:21.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:21.317+0000 7f3e51c23700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e30011db0 0x7f3e30008d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:14:22.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:22.118+0000 7f3e52424700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30010cf0 msgr2=0x7f3e300088a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:14:22.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:22.118+0000 7f3e52424700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30010cf0 0x7f3e300088a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:14:22.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:22.118+0000 7f3e51c23700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e30011db0 msgr2=0x7f3e30008d90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:14:22.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:22.118+0000 7f3e51c23700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e30011db0 0x7f3e30008d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:14:23.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:23.715+0000 7f3e43fff700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30010cf0 msgr2=0x7f3e300088a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:14:23.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:23.715+0000 7f3e43fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30010cf0 0x7f3e300088a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:14:23.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:23.715+0000 7f3e43fff700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e30011db0 msgr2=0x7f3e30008d90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:14:23.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:23.715+0000 7f3e43fff700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e30011db0 0x7f3e30008d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:14:23.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:23.715+0000 7f3e52c25700 1 -- reap_dead start 2023-10-18T16:14:23.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:23.715+0000 7f3e43fff700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e3000a820 0x7f3e30009d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:14:23.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:23.715+0000 7f3e43fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e3000e1d0 0x7f3e30012b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:14:23.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:23.715+0000 7f3e43fff700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e300129a0 con 0x7f3e30010cf0 2023-10-18T16:14:23.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:23.715+0000 7f3e43fff700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e4c14e460 con 0x7f3e3000a820 2023-10-18T16:14:23.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:23.715+0000 7f3e43fff700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4c14d650 con 0x7f3e3000e1d0 2023-10-18T16:14:23.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:23.715+0000 7f3e52424700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e3000a820 msgr2=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:14:23.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:23.715+0000 7f3e52c25700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e3000e1d0 msgr2=0x7f3e30012b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:14:23.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:23.715+0000 7f3e51c23700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e30010cf0 legacy=0x7f3e30008880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:14:23.728 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:23.715+0000 7f3e52424700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e3000a820 0x7f3e30009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:14:23.728 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:23.715+0000 7f3e52c25700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e3000e1d0 0x7f3e30012b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:14:23.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:23.916+0000 7f3e52c25700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e3000e1d0 msgr2=0x7f3e30012b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:14:23.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:23.916+0000 7f3e52424700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e3000a820 msgr2=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:14:23.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:23.916+0000 7f3e51c23700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e30010cf0 legacy=0x7f3e30008880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:14:23.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:23.916+0000 7f3e52424700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e3000a820 0x7f3e30009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:14:23.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:23.916+0000 7f3e52c25700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e3000e1d0 0x7f3e30012b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:14:24.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:24.317+0000 7f3e52424700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e3000a820 msgr2=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:14:24.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:24.317+0000 7f3e51c23700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e30010cf0 legacy=0x7f3e30008880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:14:24.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:24.317+0000 7f3e52c25700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e3000e1d0 msgr2=0x7f3e30012b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:14:24.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:24.317+0000 7f3e52424700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e3000a820 0x7f3e30009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:14:24.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:24.317+0000 7f3e52c25700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e3000e1d0 0x7f3e30012b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:14:25.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:25.118+0000 7f3e52424700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e3000a820 msgr2=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:14:25.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:25.118+0000 7f3e52424700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e3000a820 0x7f3e30009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:14:25.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:25.118+0000 7f3e51c23700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e30010cf0 legacy=0x7f3e30008880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:14:25.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:25.118+0000 7f3e52c25700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e3000e1d0 msgr2=0x7f3e30012b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:14:25.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:25.118+0000 7f3e52c25700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e3000e1d0 0x7f3e30012b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:14:26.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:26.716+0000 7f3e43fff700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e3000e1d0 msgr2=0x7f3e30012b30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:14:26.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:26.716+0000 7f3e43fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e3000e1d0 0x7f3e30012b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:14:26.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:26.716+0000 7f3e43fff700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e3000a820 msgr2=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:14:26.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:26.716+0000 7f3e43fff700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e3000a820 0x7f3e30009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:14:26.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:26.716+0000 7f3e43fff700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e30010cf0 legacy=0x7f3e30008880 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:14:26.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:26.716+0000 7f3e43fff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30013020 0x7f3e30015410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:14:26.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:26.716+0000 7f3e43fff700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e4c0b6650 con 0x7f3e30015900 2023-10-18T16:14:26.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:26.716+0000 7f3e43fff700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e4c14e460 con 0x7f3e30013020 2023-10-18T16:14:26.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:26.716+0000 7f3e52424700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30013020 msgr2=0x7f3e30015410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:14:26.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:26.716+0000 7f3e51c23700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30015900 legacy=0x7f3e30017cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:14:26.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:26.716+0000 7f3e52424700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30013020 0x7f3e30015410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:14:26.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:26.917+0000 7f3e51c23700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30015900 legacy=0x7f3e30017cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:14:26.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:26.917+0000 7f3e52424700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30013020 msgr2=0x7f3e30015410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:14:26.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:26.917+0000 7f3e52424700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30013020 0x7f3e30015410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:14:27.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:27.318+0000 7f3e52424700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30013020 msgr2=0x7f3e30015410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:14:27.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:27.318+0000 7f3e51c23700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30015900 legacy=0x7f3e30017cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:14:27.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:27.318+0000 7f3e52424700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30013020 0x7f3e30015410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:14:28.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:28.119+0000 7f3e52424700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30013020 msgr2=0x7f3e30015410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:14:28.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:28.119+0000 7f3e52424700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30013020 0x7f3e30015410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:14:28.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:28.119+0000 7f3e51c23700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30015900 legacy=0x7f3e30017cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:14:29.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:29.716+0000 7f3e43fff700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30013020 msgr2=0x7f3e30015410 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:14:29.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:29.716+0000 7f3e43fff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30013020 0x7f3e30015410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:14:29.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:29.716+0000 7f3e43fff700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30015900 legacy=0x7f3e30017cf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:14:29.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:29.716+0000 7f3e52c25700 1 -- reap_dead start 2023-10-18T16:14:29.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:29.716+0000 7f3e43fff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30013020 0x7f3e30015410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:14:29.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:29.716+0000 7f3e52424700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30013020 msgr2=0x7f3e30015410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:14:29.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:29.716+0000 7f3e43fff700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e300129a0 con 0x7f3e3000a820 2023-10-18T16:14:29.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:29.716+0000 7f3e43fff700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e4c0b6650 con 0x7f3e30010cf0 2023-10-18T16:14:29.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:29.716+0000 7f3e43fff700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e4c14e460 con 0x7f3e30013020 2023-10-18T16:14:29.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:29.716+0000 7f3e52424700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30013020 0x7f3e30015410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:14:29.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:29.716+0000 7f3e51c23700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e3000a820 legacy=0x7f3e30008880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:14:29.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:29.716+0000 7f3e52c25700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e30010cf0 legacy=0x7f3e30003970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:14:29.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:29.917+0000 7f3e52424700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30013020 msgr2=0x7f3e30015410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:14:29.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:29.917+0000 7f3e51c23700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e3000a820 legacy=0x7f3e30008880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:14:29.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:29.917+0000 7f3e52424700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30013020 0x7f3e30015410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:14:29.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:29.917+0000 7f3e52c25700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e30010cf0 legacy=0x7f3e30003970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:14:30.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:30.317+0000 7f3e52424700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30013020 msgr2=0x7f3e30015410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:14:30.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:30.317+0000 7f3e51c23700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e3000a820 legacy=0x7f3e30008880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:14:30.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:30.317+0000 7f3e52424700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30013020 0x7f3e30015410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:14:30.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:30.317+0000 7f3e52c25700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e30010cf0 legacy=0x7f3e30003970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:14:31.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:31.118+0000 7f3e51c23700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e3000a820 legacy=0x7f3e30008880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:14:31.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:31.118+0000 7f3e52424700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30013020 msgr2=0x7f3e30015410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:14:31.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:31.118+0000 7f3e52424700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30013020 0x7f3e30015410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:14:31.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:31.118+0000 7f3e52c25700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e30010cf0 legacy=0x7f3e30003970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:14:32.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:32.715+0000 7f3e43fff700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30013020 msgr2=0x7f3e30015410 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:14:32.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:32.715+0000 7f3e43fff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30013020 0x7f3e30015410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:14:32.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:32.715+0000 7f3e43fff700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e30010cf0 legacy=0x7f3e30003970 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:14:32.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:32.715+0000 7f3e43fff700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e3000a820 legacy=0x7f3e30008880 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:14:32.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:32.715+0000 7f3e43fff700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e3000e1d0 0x7f3e30009d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:14:32.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:32.715+0000 7f3e43fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30012b30 0x7f3e30005090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:14:32.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:32.715+0000 7f3e43fff700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e4c14e460 con 0x7f3e3000e1d0 2023-10-18T16:14:32.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:32.715+0000 7f3e43fff700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4c14d650 con 0x7f3e30012b30 2023-10-18T16:14:32.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:32.715+0000 7f3e52424700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e3000e1d0 msgr2=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:14:32.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:32.715+0000 7f3e51c23700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30012b30 msgr2=0x7f3e30005090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:14:32.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:32.715+0000 7f3e52424700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e3000e1d0 0x7f3e30009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:14:32.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:32.715+0000 7f3e51c23700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30012b30 0x7f3e30005090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:14:32.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:32.916+0000 7f3e52424700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e3000e1d0 msgr2=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:14:32.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:32.916+0000 7f3e52424700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e3000e1d0 0x7f3e30009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:14:32.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:32.917+0000 7f3e51c23700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30012b30 msgr2=0x7f3e30005090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:14:32.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:32.917+0000 7f3e51c23700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30012b30 0x7f3e30005090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:14:33.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:33.317+0000 7f3e52424700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e3000e1d0 msgr2=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:14:33.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:33.317+0000 7f3e52424700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e3000e1d0 0x7f3e30009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:14:33.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:33.318+0000 7f3e51c23700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30012b30 msgr2=0x7f3e30005090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:14:33.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:33.318+0000 7f3e51c23700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30012b30 0x7f3e30005090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:14:34.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:34.118+0000 7f3e52424700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e3000e1d0 msgr2=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:14:34.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:34.118+0000 7f3e52424700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e3000e1d0 0x7f3e30009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:14:34.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:34.119+0000 7f3e51c23700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30012b30 msgr2=0x7f3e30005090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:14:34.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:34.119+0000 7f3e51c23700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30012b30 0x7f3e30005090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:14:35.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:35.715+0000 7f3e43fff700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30012b30 msgr2=0x7f3e30005090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:14:35.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:35.715+0000 7f3e43fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30012b30 0x7f3e30005090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:14:35.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:35.715+0000 7f3e43fff700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e3000e1d0 msgr2=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:14:35.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:35.715+0000 7f3e43fff700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e3000e1d0 0x7f3e30009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:14:35.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:35.715+0000 7f3e52c25700 1 -- reap_dead start 2023-10-18T16:14:35.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:35.715+0000 7f3e43fff700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e3000a820 0x7f3e30009d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:14:35.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:35.715+0000 7f3e43fff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30010cf0 0x7f3e30008880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:14:35.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:35.715+0000 7f3e43fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30012b30 0x7f3e3000f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:14:35.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:35.716+0000 7f3e43fff700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e4c14e460 con 0x7f3e30010cf0 2023-10-18T16:14:35.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:35.716+0000 7f3e43fff700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e4c14d650 con 0x7f3e3000a820 2023-10-18T16:14:35.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:35.716+0000 7f3e43fff700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e3000ac00 con 0x7f3e30012b30 2023-10-18T16:14:35.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:35.716+0000 7f3e51c23700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30010cf0 msgr2=0x7f3e30008880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:14:35.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:35.716+0000 7f3e51c23700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30010cf0 0x7f3e30008880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:14:35.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:35.716+0000 7f3e52424700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e3000a820 msgr2=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:14:35.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:35.716+0000 7f3e52c25700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30012b30 msgr2=0x7f3e3000f1d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:14:35.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:35.716+0000 7f3e52424700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e3000a820 0x7f3e30009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:14:35.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:35.716+0000 7f3e52c25700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30012b30 0x7f3e3000f1d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:14:35.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:35.917+0000 7f3e51c23700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30010cf0 msgr2=0x7f3e30008880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:14:35.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:35.917+0000 7f3e51c23700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30010cf0 0x7f3e30008880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:14:35.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:35.917+0000 7f3e52c25700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30012b30 msgr2=0x7f3e3000f1d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:14:35.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:35.917+0000 7f3e52424700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e3000a820 msgr2=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:14:35.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:35.917+0000 7f3e52c25700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30012b30 0x7f3e3000f1d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:14:35.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:35.917+0000 7f3e52424700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e3000a820 0x7f3e30009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:14:36.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:36.318+0000 7f3e51c23700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30010cf0 msgr2=0x7f3e30008880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:14:36.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:36.318+0000 7f3e51c23700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30010cf0 0x7f3e30008880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:14:36.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:36.318+0000 7f3e52c25700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30012b30 msgr2=0x7f3e3000f1d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:14:36.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:36.318+0000 7f3e52424700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e3000a820 msgr2=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:14:36.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:36.318+0000 7f3e52424700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e3000a820 0x7f3e30009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:14:36.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:36.318+0000 7f3e52c25700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30012b30 0x7f3e3000f1d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:14:37.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:37.119+0000 7f3e51c23700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30010cf0 msgr2=0x7f3e30008880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:14:37.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:37.119+0000 7f3e51c23700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30010cf0 0x7f3e30008880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:14:37.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:37.119+0000 7f3e52424700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e3000a820 msgr2=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:14:37.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:37.119+0000 7f3e52424700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e3000a820 0x7f3e30009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:14:37.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:37.119+0000 7f3e52c25700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30012b30 msgr2=0x7f3e3000f1d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:14:37.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:37.119+0000 7f3e52c25700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30012b30 0x7f3e3000f1d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:14:38.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:38.716+0000 7f3e43fff700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30012b30 msgr2=0x7f3e3000f1d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:14:38.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:38.716+0000 7f3e43fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30012b30 0x7f3e3000f1d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:14:38.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:38.716+0000 7f3e43fff700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e3000a820 msgr2=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:14:38.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:38.716+0000 7f3e43fff700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e3000a820 0x7f3e30009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:14:38.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:38.716+0000 7f3e43fff700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30010cf0 msgr2=0x7f3e30008880 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:14:38.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:38.716+0000 7f3e43fff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30010cf0 0x7f3e30008880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:14:38.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:38.716+0000 7f3e43fff700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e4c05e0f0 con 0x7f3e300101b0 2023-10-18T16:14:38.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:38.716+0000 7f3e43fff700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e300129a0 con 0x7f3e3000f6c0 2023-10-18T16:14:38.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:38.716+0000 7f3e51c23700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e300101b0 legacy=0x7f3e30018050 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:14:38.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:38.716+0000 7f3e52424700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e3000f6c0 legacy=0x7f3e3000faa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:14:38.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:38.917+0000 7f3e51c23700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e300101b0 legacy=0x7f3e30018050 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:14:38.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:38.917+0000 7f3e52424700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e3000f6c0 legacy=0x7f3e3000faa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:14:39.154 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-18T16:14:39.157 DEBUG:teuthology.orchestra.run.smithi151:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-18T16:14:39.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:39.318+0000 7f3e51c23700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e300101b0 legacy=0x7f3e30018050 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:14:39.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:39.318+0000 7f3e52424700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e3000f6c0 legacy=0x7f3e3000faa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:14:40.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:40.119+0000 7f3e51c23700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e300101b0 legacy=0x7f3e30018050 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:14:40.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:40.119+0000 7f3e52424700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e3000f6c0 legacy=0x7f3e3000faa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:14:41.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:41.716+0000 7f3e43fff700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e3000f6c0 legacy=0x7f3e3000faa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:14:41.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:41.716+0000 7f3e43fff700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e300101b0 legacy=0x7f3e30018050 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:14:41.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:41.716+0000 7f3e52c25700 1 -- reap_dead start 2023-10-18T16:14:41.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:41.716+0000 7f3e43fff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30010cf0 0x7f3e30008880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:14:41.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:41.716+0000 7f3e43fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e3000a820 0x7f3e30009d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:14:41.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:41.716+0000 7f3e52424700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30010cf0 msgr2=0x7f3e30008880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:14:41.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:41.716+0000 7f3e43fff700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e4c0b6650 con 0x7f3e30012b30 2023-10-18T16:14:41.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:41.716+0000 7f3e43fff700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e4c14e460 con 0x7f3e30010cf0 2023-10-18T16:14:41.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:41.716+0000 7f3e52424700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30010cf0 0x7f3e30008880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:14:41.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:41.716+0000 7f3e43fff700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4c14d650 con 0x7f3e3000a820 2023-10-18T16:14:41.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:41.716+0000 7f3e52c25700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30012b30 legacy=0x7f3e3000f1d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:14:41.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:41.716+0000 7f3e51c23700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e3000a820 msgr2=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:14:41.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:41.716+0000 7f3e51c23700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e3000a820 0x7f3e30009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:14:41.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:41.917+0000 7f3e52424700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30010cf0 msgr2=0x7f3e30008880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:14:41.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:41.917+0000 7f3e52424700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30010cf0 0x7f3e30008880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:14:41.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:41.917+0000 7f3e52c25700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30012b30 legacy=0x7f3e3000f1d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:14:41.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:41.917+0000 7f3e51c23700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e3000a820 msgr2=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:14:41.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:41.917+0000 7f3e51c23700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e3000a820 0x7f3e30009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:14:42.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:42.318+0000 7f3e52c25700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30012b30 legacy=0x7f3e3000f1d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:14:42.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:42.318+0000 7f3e52424700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30010cf0 msgr2=0x7f3e30008880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:14:42.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:42.318+0000 7f3e52424700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30010cf0 0x7f3e30008880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:14:42.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:42.318+0000 7f3e51c23700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e3000a820 msgr2=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:14:42.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:42.318+0000 7f3e51c23700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e3000a820 0x7f3e30009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:14:43.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:43.119+0000 7f3e52c25700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30012b30 legacy=0x7f3e3000f1d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:14:43.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:43.119+0000 7f3e52424700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30010cf0 msgr2=0x7f3e30008880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:14:43.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:43.119+0000 7f3e52424700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30010cf0 0x7f3e30008880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:14:43.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:43.119+0000 7f3e51c23700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e3000a820 msgr2=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:14:43.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:43.119+0000 7f3e51c23700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e3000a820 0x7f3e30009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:14:44.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:44.717+0000 7f3e43fff700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e3000a820 msgr2=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:14:44.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:44.717+0000 7f3e43fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e3000a820 0x7f3e30009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:14:44.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:44.717+0000 7f3e43fff700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e30010cf0 msgr2=0x7f3e30008880 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:14:44.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:44.717+0000 7f3e43fff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e30010cf0 0x7f3e30008880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:14:44.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:44.717+0000 7f3e43fff700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30012b30 legacy=0x7f3e3000f1d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:14:44.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:44.717+0000 7f3e43fff700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e30008d70 0x7f3e3000f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:14:44.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:44.717+0000 7f3e43fff700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e4c05e0f0 con 0x7f3e3000fdd0 2023-10-18T16:14:44.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:44.717+0000 7f3e43fff700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e4c14e460 con 0x7f3e30008d70 2023-10-18T16:14:44.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:44.717+0000 7f3e52424700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e30008d70 msgr2=0x7f3e3000f8e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:14:44.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:44.717+0000 7f3e51c23700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e3000fdd0 legacy=0x7f3e30018030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:14:44.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:44.717+0000 7f3e52424700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e30008d70 0x7f3e3000f8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:14:44.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:44.918+0000 7f3e51c23700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e3000fdd0 legacy=0x7f3e30018030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:14:44.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:44.918+0000 7f3e52424700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e30008d70 msgr2=0x7f3e3000f8e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:14:44.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:44.918+0000 7f3e52424700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e30008d70 0x7f3e3000f8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:14:45.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:45.319+0000 7f3e52424700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e30008d70 msgr2=0x7f3e3000f8e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:14:45.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:45.319+0000 7f3e51c23700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e3000fdd0 legacy=0x7f3e30018030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:14:45.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:45.319+0000 7f3e52424700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e30008d70 0x7f3e3000f8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:14:46.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:46.120+0000 7f3e52424700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e30008d70 msgr2=0x7f3e3000f8e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:14:46.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:46.120+0000 7f3e51c23700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e3000fdd0 legacy=0x7f3e30018030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:14:46.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:46.120+0000 7f3e52424700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e30008d70 0x7f3e3000f8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:14:47.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:47.717+0000 7f3e43fff700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e30008d70 msgr2=0x7f3e3000f8e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:14:47.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:47.717+0000 7f3e43fff700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e30008d70 0x7f3e3000f8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:14:47.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:47.717+0000 7f3e43fff700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e3000fdd0 legacy=0x7f3e30018030 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:14:47.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:47.717+0000 7f3e43fff700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e30008d70 0x7f3e3000f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:14:47.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:47.717+0000 7f3e52c25700 1 -- reap_dead start 2023-10-18T16:14:47.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:47.717+0000 7f3e43fff700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e4c0b6650 con 0x7f3e30012b30 2023-10-18T16:14:47.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:47.717+0000 7f3e43fff700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e4c05e0f0 con 0x7f3e3000fdd0 2023-10-18T16:14:47.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:47.717+0000 7f3e43fff700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e4c14e460 con 0x7f3e30008d70 2023-10-18T16:14:47.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:47.717+0000 7f3e52424700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30012b30 legacy=0x7f3e3000f1d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:14:47.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:47.717+0000 7f3e51c23700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e30008d70 msgr2=0x7f3e3000f8e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:14:47.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:47.717+0000 7f3e52c25700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e3000fdd0 legacy=0x7f3e30018030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:14:47.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:47.717+0000 7f3e51c23700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e30008d70 0x7f3e3000f8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:14:47.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:47.918+0000 7f3e52424700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30012b30 legacy=0x7f3e3000f1d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:14:47.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:47.918+0000 7f3e52c25700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e3000fdd0 legacy=0x7f3e30018030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:14:47.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:47.918+0000 7f3e51c23700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e30008d70 msgr2=0x7f3e3000f8e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:14:47.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:47.918+0000 7f3e51c23700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e30008d70 0x7f3e3000f8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:14:48.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:48.319+0000 7f3e52424700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30012b30 legacy=0x7f3e3000f1d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:14:48.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:48.319+0000 7f3e52c25700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e3000fdd0 legacy=0x7f3e30018030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:14:48.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:48.319+0000 7f3e51c23700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e30008d70 msgr2=0x7f3e3000f8e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:14:48.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:48.319+0000 7f3e51c23700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e30008d70 0x7f3e3000f8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:14:49.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:49.120+0000 7f3e52424700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30012b30 legacy=0x7f3e3000f1d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:14:49.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:49.120+0000 7f3e51c23700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e30008d70 msgr2=0x7f3e3000f8e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:14:49.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:49.120+0000 7f3e52c25700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e3000fdd0 legacy=0x7f3e30018030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:14:49.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:49.120+0000 7f3e51c23700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e30008d70 0x7f3e3000f8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:14:50.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:50.717+0000 7f3e43fff700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f3e30008d70 msgr2=0x7f3e3000f8e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:14:50.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:50.717+0000 7f3e43fff700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3e30008d70 0x7f3e3000f8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:14:50.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:50.717+0000 7f3e43fff700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f3e3000fdd0 legacy=0x7f3e30018030 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:14:50.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:50.717+0000 7f3e43fff700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f3e30012b30 legacy=0x7f3e3000f1d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:14:50.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:50.717+0000 7f3e43fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30010cf0 0x7f3e30008880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:14:50.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:50.717+0000 7f3e43fff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e3000a820 0x7f3e30009d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:14:50.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:50.717+0000 7f3e43fff700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e4c14e460 con 0x7f3e3000a820 2023-10-18T16:14:50.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:50.717+0000 7f3e43fff700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4c14d650 con 0x7f3e30010cf0 2023-10-18T16:14:50.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:50.717+0000 7f3e51c23700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e3000a820 msgr2=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:14:50.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:50.717+0000 7f3e51c23700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e3000a820 0x7f3e30009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:14:50.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:50.717+0000 7f3e52424700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30010cf0 msgr2=0x7f3e30008880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:14:50.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:50.717+0000 7f3e52424700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30010cf0 0x7f3e30008880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:14:50.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:50.918+0000 7f3e51c23700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e3000a820 msgr2=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:14:50.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:50.918+0000 7f3e51c23700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e3000a820 0x7f3e30009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:14:50.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:50.918+0000 7f3e52424700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30010cf0 msgr2=0x7f3e30008880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:14:50.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:50.918+0000 7f3e52424700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30010cf0 0x7f3e30008880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:14:51.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:51.319+0000 7f3e51c23700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e3000a820 msgr2=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:14:51.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:51.319+0000 7f3e51c23700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e3000a820 0x7f3e30009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:14:51.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:51.319+0000 7f3e52424700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30010cf0 msgr2=0x7f3e30008880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:14:51.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:51.319+0000 7f3e52424700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30010cf0 0x7f3e30008880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:14:52.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:52.120+0000 7f3e51c23700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f3e3000a820 msgr2=0x7f3e30009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:14:52.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:52.120+0000 7f3e51c23700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e3000a820 0x7f3e30009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:14:52.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:52.120+0000 7f3e52424700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30010cf0 msgr2=0x7f3e30008880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:14:52.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:52.120+0000 7f3e52424700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3e30010cf0 0x7f3e30008880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:14:53.618 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-10-18T16:14:53.619 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2023-10-18T16:14:53.620 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-18T16:14:53.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:53.736+0000 7fcf6f314700 1 Processor -- start 2023-10-18T16:14:53.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:53.737+0000 7fcf6f314700 1 -- start start 2023-10-18T16:14:53.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:53.737+0000 7fcf6f314700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf680c0210 0x7fcf68151370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:14:53.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:53.737+0000 7fcf6f314700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf680bef50 0x7fcf680c05f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:14:53.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:53.737+0000 7fcf6f314700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf680be360 con 0x7fcf6814dcc0 2023-10-18T16:14:53.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:53.737+0000 7fcf6f314700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf680bad60 con 0x7fcf680c0210 2023-10-18T16:14:53.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:53.737+0000 7fcf6f314700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf6814d2b0 con 0x7fcf680bef50 2023-10-18T16:14:53.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:53.737+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf680c0210 msgr2=0x7fcf68151370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:14:53.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:53.737+0000 7fcf6d8b1700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf6814dcc0 legacy=0x7fcf68143c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:14:53.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:53.737+0000 7fcf6c8af700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf680bef50 msgr2=0x7fcf680c05f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:14:53.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:53.737+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf680c0210 0x7fcf68151370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:14:53.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:53.738+0000 7fcf6c8af700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf680bef50 0x7fcf680c05f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:14:53.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:53.938+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf680c0210 msgr2=0x7fcf68151370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:14:53.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:53.938+0000 7fcf6d8b1700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf6814dcc0 legacy=0x7fcf68143c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:14:53.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:53.938+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf680c0210 0x7fcf68151370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:14:53.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:53.939+0000 7fcf6c8af700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf680bef50 msgr2=0x7fcf680c05f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:14:53.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:53.939+0000 7fcf6c8af700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf680bef50 0x7fcf680c05f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:14:54.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:54.339+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf680c0210 msgr2=0x7fcf68151370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:14:54.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:54.339+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf680c0210 0x7fcf68151370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:14:54.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:54.339+0000 7fcf6d8b1700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf6814dcc0 legacy=0x7fcf68143c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:14:54.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:54.340+0000 7fcf6c8af700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf680bef50 msgr2=0x7fcf680c05f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:14:54.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:54.340+0000 7fcf6c8af700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf680bef50 0x7fcf680c05f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:14:55.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:55.140+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf680c0210 msgr2=0x7fcf68151370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:14:55.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:55.140+0000 7fcf6d8b1700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf6814dcc0 legacy=0x7fcf68143c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:14:55.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:55.140+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf680c0210 0x7fcf68151370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:14:55.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:55.141+0000 7fcf6c8af700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf680bef50 msgr2=0x7fcf680c05f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:14:55.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:55.141+0000 7fcf6c8af700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf680bef50 0x7fcf680c05f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:14:56.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:56.737+0000 7fcf5effd700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf680bef50 msgr2=0x7fcf680c05f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:14:56.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:56.737+0000 7fcf5effd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf680bef50 0x7fcf680c05f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:14:56.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:56.737+0000 7fcf5effd700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf680c0210 msgr2=0x7fcf68151370 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:14:56.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:56.737+0000 7fcf5effd700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf680c0210 0x7fcf68151370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:14:56.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:56.737+0000 7fcf5effd700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf6814dcc0 legacy=0x7fcf68143c10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:14:56.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:56.738+0000 7fcf5effd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c003590 0x7fcf4c005a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:14:56.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:56.738+0000 7fcf5effd700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf680be640 con 0x7fcf4c006010 2023-10-18T16:14:56.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:56.738+0000 7fcf5effd700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf6814d2b0 con 0x7fcf4c003590 2023-10-18T16:14:56.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:56.738+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c003590 msgr2=0x7fcf4c005a10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:14:56.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:56.738+0000 7fcf6c8af700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c006010 legacy=0x7fcf4c0084d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:14:56.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:56.738+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c003590 0x7fcf4c005a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:14:56.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:56.939+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c003590 msgr2=0x7fcf4c005a10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:14:56.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:56.939+0000 7fcf6c8af700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c006010 legacy=0x7fcf4c0084d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:14:56.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:56.939+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c003590 0x7fcf4c005a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:14:57.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:57.340+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c003590 msgr2=0x7fcf4c005a10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:14:57.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:57.340+0000 7fcf6c8af700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c006010 legacy=0x7fcf4c0084d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:14:57.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:57.340+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c003590 0x7fcf4c005a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:14:58.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:58.141+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c003590 msgr2=0x7fcf4c005a10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:14:58.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:58.141+0000 7fcf6c8af700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c006010 legacy=0x7fcf4c0084d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:14:58.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:58.141+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c003590 0x7fcf4c005a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:14:59.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:59.738+0000 7fcf5effd700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c003590 msgr2=0x7fcf4c005a10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:14:59.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:59.738+0000 7fcf5effd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c003590 0x7fcf4c005a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:14:59.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:59.738+0000 7fcf5effd700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c006010 legacy=0x7fcf4c0084d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:14:59.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:59.738+0000 7fcf6d8b1700 1 -- reap_dead start 2023-10-18T16:14:59.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:59.738+0000 7fcf5effd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf680bef50 0x7fcf4c009d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:14:59.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:59.738+0000 7fcf5effd700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c005ab0 0x7fcf4c0084d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:14:59.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:59.738+0000 7fcf5effd700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf680be360 con 0x7fcf4c00a270 2023-10-18T16:14:59.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:59.738+0000 7fcf6c8af700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c00a270 legacy=0x7fcf4c00a650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:14:59.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:59.738+0000 7fcf5effd700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf6814d2b0 con 0x7fcf4c005ab0 2023-10-18T16:14:59.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:59.738+0000 7fcf5effd700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf4c010160 con 0x7fcf680bef50 2023-10-18T16:14:59.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:59.738+0000 7fcf6d0b0700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf680bef50 msgr2=0x7fcf4c009d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:14:59.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:59.738+0000 7fcf6d0b0700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf680bef50 0x7fcf4c009d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:14:59.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:59.738+0000 7fcf6d8b1700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c005ab0 msgr2=0x7fcf4c0084d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:14:59.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:59.738+0000 7fcf6d8b1700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c005ab0 0x7fcf4c0084d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:14:59.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:59.939+0000 7fcf6c8af700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c00a270 legacy=0x7fcf4c00a650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:14:59.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:59.939+0000 7fcf6d0b0700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf680bef50 msgr2=0x7fcf4c009d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:14:59.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:59.939+0000 7fcf6d8b1700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c005ab0 msgr2=0x7fcf4c0084d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:14:59.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:59.939+0000 7fcf6d0b0700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf680bef50 0x7fcf4c009d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:14:59.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:14:59.939+0000 7fcf6d8b1700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c005ab0 0x7fcf4c0084d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:15:00.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:00.340+0000 7fcf6c8af700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c00a270 legacy=0x7fcf4c00a650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:15:00.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:00.340+0000 7fcf6d0b0700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf680bef50 msgr2=0x7fcf4c009d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:15:00.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:00.340+0000 7fcf6d8b1700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c005ab0 msgr2=0x7fcf4c0084d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:15:00.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:00.340+0000 7fcf6d0b0700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf680bef50 0x7fcf4c009d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:15:00.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:00.340+0000 7fcf6d8b1700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c005ab0 0x7fcf4c0084d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:15:01.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:01.141+0000 7fcf6c8af700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c00a270 legacy=0x7fcf4c00a650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:15:01.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:01.141+0000 7fcf6d0b0700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf680bef50 msgr2=0x7fcf4c009d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:15:01.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:01.141+0000 7fcf6d8b1700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c005ab0 msgr2=0x7fcf4c0084d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:15:01.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:01.141+0000 7fcf6d0b0700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf680bef50 0x7fcf4c009d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:15:01.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:01.141+0000 7fcf6d8b1700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c005ab0 0x7fcf4c0084d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:15:02.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:02.739+0000 7fcf5effd700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf680bef50 msgr2=0x7fcf4c009d60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:15:02.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:02.739+0000 7fcf5effd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf680bef50 0x7fcf4c009d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:15:02.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:02.739+0000 7fcf5effd700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c005ab0 msgr2=0x7fcf4c0084d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:15:02.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:02.739+0000 7fcf5effd700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c005ab0 0x7fcf4c0084d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:15:02.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:02.739+0000 7fcf5effd700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c00a270 legacy=0x7fcf4c00a650 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:15:02.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:02.739+0000 7fcf5effd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c013310 0x7fcf4c015700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:15:02.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:02.739+0000 7fcf5effd700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf680be640 con 0x7fcf4c00f180 2023-10-18T16:15:02.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:02.739+0000 7fcf5effd700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf6814d2b0 con 0x7fcf4c013310 2023-10-18T16:15:02.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:02.739+0000 7fcf6c8af700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c013310 msgr2=0x7fcf4c015700 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:15:02.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:02.739+0000 7fcf6c8af700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c013310 0x7fcf4c015700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:15:02.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:02.739+0000 7fcf6d0b0700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c00f180 legacy=0x7fcf4c00f560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:15:02.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:02.940+0000 7fcf6c8af700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c013310 msgr2=0x7fcf4c015700 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:15:02.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:02.940+0000 7fcf6c8af700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c013310 0x7fcf4c015700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:15:02.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:02.940+0000 7fcf6d0b0700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c00f180 legacy=0x7fcf4c00f560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:15:03.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:03.341+0000 7fcf6c8af700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c013310 msgr2=0x7fcf4c015700 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:15:03.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:03.341+0000 7fcf6c8af700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c013310 0x7fcf4c015700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:15:03.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:03.341+0000 7fcf6d0b0700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c00f180 legacy=0x7fcf4c00f560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:15:04.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:04.142+0000 7fcf6c8af700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c013310 msgr2=0x7fcf4c015700 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:15:04.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:04.142+0000 7fcf6c8af700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c013310 0x7fcf4c015700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:15:04.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:04.142+0000 7fcf6d0b0700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c00f180 legacy=0x7fcf4c00f560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:15:05.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:05.739+0000 7fcf5effd700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c013310 msgr2=0x7fcf4c015700 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:15:05.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:05.739+0000 7fcf5effd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c013310 0x7fcf4c015700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:15:05.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:05.739+0000 7fcf5effd700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c00f180 legacy=0x7fcf4c00f560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:15:05.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:05.739+0000 7fcf6d8b1700 1 -- reap_dead start 2023-10-18T16:15:05.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:05.739+0000 7fcf5effd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c00a980 0x7fcf4c0122d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:15:05.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:05.739+0000 7fcf5effd700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf680be360 con 0x7fcf4c009d30 2023-10-18T16:15:05.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:05.739+0000 7fcf6c8af700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c009d30 legacy=0x7fcf4c00a270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:15:05.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:05.739+0000 7fcf5effd700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf680be640 con 0x7fcf4c005ab0 2023-10-18T16:15:05.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:05.739+0000 7fcf5effd700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf6814d2b0 con 0x7fcf4c00a980 2023-10-18T16:15:05.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:05.739+0000 7fcf6d0b0700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c005ab0 legacy=0x7fcf4c00f560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:15:05.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:05.739+0000 7fcf6d8b1700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c00a980 msgr2=0x7fcf4c0122d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:15:05.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:05.739+0000 7fcf6d8b1700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c00a980 0x7fcf4c0122d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:15:05.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:05.940+0000 7fcf6c8af700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c009d30 legacy=0x7fcf4c00a270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:15:05.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:05.940+0000 7fcf6d0b0700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c005ab0 legacy=0x7fcf4c00f560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:15:05.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:05.940+0000 7fcf6d8b1700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c00a980 msgr2=0x7fcf4c0122d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:15:05.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:05.940+0000 7fcf6d8b1700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c00a980 0x7fcf4c0122d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:15:06.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:06.341+0000 7fcf6c8af700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c009d30 legacy=0x7fcf4c00a270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:15:06.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:06.341+0000 7fcf6d8b1700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c00a980 msgr2=0x7fcf4c0122d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:15:06.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:06.341+0000 7fcf6d0b0700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c005ab0 legacy=0x7fcf4c00f560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:15:06.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:06.341+0000 7fcf6d8b1700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c00a980 0x7fcf4c0122d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:15:07.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:07.142+0000 7fcf6c8af700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c009d30 legacy=0x7fcf4c00a270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:15:07.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:07.142+0000 7fcf6d8b1700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c00a980 msgr2=0x7fcf4c0122d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:15:07.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:07.142+0000 7fcf6d0b0700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c005ab0 legacy=0x7fcf4c00f560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:15:07.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:07.142+0000 7fcf6d8b1700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c00a980 0x7fcf4c0122d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:15:08.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:08.739+0000 7fcf5effd700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c00a980 msgr2=0x7fcf4c0122d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:15:08.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:08.739+0000 7fcf5effd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c00a980 0x7fcf4c0122d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:15:08.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:08.739+0000 7fcf5effd700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c005ab0 legacy=0x7fcf4c00f560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:15:08.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:08.739+0000 7fcf5effd700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c009d30 legacy=0x7fcf4c00a270 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:15:08.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:08.739+0000 7fcf5effd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c0127c0 0x7fcf4c012bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:15:08.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:08.739+0000 7fcf5effd700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c0074d0 0x7fcf4c007920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:15:08.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:08.739+0000 7fcf5effd700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf6814d2b0 con 0x7fcf4c0127c0 2023-10-18T16:15:08.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:08.739+0000 7fcf5effd700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf4c010160 con 0x7fcf4c0074d0 2023-10-18T16:15:08.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:08.739+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c0127c0 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:15:08.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:08.739+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c0127c0 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:15:08.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:08.739+0000 7fcf6c8af700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c0074d0 msgr2=0x7fcf4c007920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:15:08.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:08.739+0000 7fcf6c8af700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c0074d0 0x7fcf4c007920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:15:08.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:08.940+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c0127c0 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:15:08.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:08.940+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c0127c0 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:15:08.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:08.940+0000 7fcf6c8af700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c0074d0 msgr2=0x7fcf4c007920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:15:08.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:08.940+0000 7fcf6c8af700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c0074d0 0x7fcf4c007920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:15:09.240 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-18T16:15:09.243 DEBUG:teuthology.orchestra.run.smithi151:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-18T16:15:09.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:09.341+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c0127c0 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:15:09.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:09.341+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c0127c0 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:15:09.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:09.341+0000 7fcf6c8af700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c0074d0 msgr2=0x7fcf4c007920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:15:09.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:09.341+0000 7fcf6c8af700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c0074d0 0x7fcf4c007920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:15:10.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:10.142+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c0127c0 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:15:10.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:10.142+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c0127c0 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:15:10.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:10.142+0000 7fcf6c8af700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c0074d0 msgr2=0x7fcf4c007920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:15:10.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:10.142+0000 7fcf6c8af700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c0074d0 0x7fcf4c007920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:15:11.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:11.740+0000 7fcf5effd700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c0074d0 msgr2=0x7fcf4c007920 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:15:11.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:11.740+0000 7fcf5effd700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c0074d0 0x7fcf4c007920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:15:11.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:11.740+0000 7fcf5effd700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c0127c0 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:15:11.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:11.740+0000 7fcf5effd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c0127c0 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:15:11.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:11.740+0000 7fcf6d8b1700 1 -- reap_dead start 2023-10-18T16:15:11.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:11.740+0000 7fcf5effd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c009d30 0x7fcf4c00a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:15:11.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:11.740+0000 7fcf5effd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c00f940 0x7fcf4c009310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:15:11.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:11.740+0000 7fcf5effd700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf4c0084e0 con 0x7fcf4c00f560 2023-10-18T16:15:11.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:11.740+0000 7fcf5effd700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf6814d2b0 con 0x7fcf4c00f940 2023-10-18T16:15:11.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:11.740+0000 7fcf5effd700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf4c010160 con 0x7fcf4c009d30 2023-10-18T16:15:11.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:11.740+0000 7fcf6c8af700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c00f560 legacy=0x7fcf4c008c00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:15:11.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:11.740+0000 7fcf6d8b1700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c00f940 msgr2=0x7fcf4c009310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:15:11.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:11.740+0000 7fcf6d8b1700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c00f940 0x7fcf4c009310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:15:11.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:11.740+0000 7fcf6d0b0700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c009d30 msgr2=0x7fcf4c00a270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:15:11.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:11.740+0000 7fcf6d0b0700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c009d30 0x7fcf4c00a270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:15:11.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:11.941+0000 7fcf6c8af700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c00f560 legacy=0x7fcf4c008c00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:15:11.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:11.941+0000 7fcf6d8b1700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c00f940 msgr2=0x7fcf4c009310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:15:11.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:11.941+0000 7fcf6d8b1700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c00f940 0x7fcf4c009310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:15:11.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:11.941+0000 7fcf6d0b0700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c009d30 msgr2=0x7fcf4c00a270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:15:11.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:11.941+0000 7fcf6d0b0700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c009d30 0x7fcf4c00a270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:15:12.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:12.342+0000 7fcf6d8b1700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c00f940 msgr2=0x7fcf4c009310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:15:12.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:12.342+0000 7fcf6c8af700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c00f560 legacy=0x7fcf4c008c00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:15:12.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:12.342+0000 7fcf6d8b1700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c00f940 0x7fcf4c009310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:15:12.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:12.342+0000 7fcf6d0b0700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c009d30 msgr2=0x7fcf4c00a270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:15:12.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:12.342+0000 7fcf6d0b0700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c009d30 0x7fcf4c00a270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:15:13.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:13.143+0000 7fcf6c8af700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c00f560 legacy=0x7fcf4c008c00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:15:13.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:13.143+0000 7fcf6d8b1700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c00f940 msgr2=0x7fcf4c009310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:15:13.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:13.143+0000 7fcf6d8b1700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c00f940 0x7fcf4c009310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:15:13.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:13.143+0000 7fcf6d0b0700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c009d30 msgr2=0x7fcf4c00a270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:15:13.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:13.143+0000 7fcf6d0b0700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c009d30 0x7fcf4c00a270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:15:14.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:14.740+0000 7fcf5effd700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c009d30 msgr2=0x7fcf4c00a270 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:15:14.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:14.740+0000 7fcf5effd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c009d30 0x7fcf4c00a270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:15:14.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:14.740+0000 7fcf5effd700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c00f940 msgr2=0x7fcf4c009310 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:15:14.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:14.740+0000 7fcf5effd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c00f940 0x7fcf4c009310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:15:14.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:14.740+0000 7fcf5effd700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c00f560 legacy=0x7fcf4c008c00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:15:14.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:14.740+0000 7fcf5effd700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c009800 0x7fcf4c0122d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:15:14.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:14.740+0000 7fcf5effd700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf680be360 con 0x7fcf4c0127c0 2023-10-18T16:15:14.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:14.740+0000 7fcf5effd700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf6814d2b0 con 0x7fcf4c009800 2023-10-18T16:15:14.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:14.740+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c009800 msgr2=0x7fcf4c0122d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:15:14.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:14.740+0000 7fcf6c8af700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c0127c0 legacy=0x7fcf4c004a10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:15:14.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:14.740+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c009800 0x7fcf4c0122d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:15:14.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:14.941+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c009800 msgr2=0x7fcf4c0122d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:15:14.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:14.941+0000 7fcf6c8af700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c0127c0 legacy=0x7fcf4c004a10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:15:14.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:14.941+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c009800 0x7fcf4c0122d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:15:15.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:15.342+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c009800 msgr2=0x7fcf4c0122d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:15:15.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:15.342+0000 7fcf6c8af700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c0127c0 legacy=0x7fcf4c004a10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:15:15.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:15.342+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c009800 0x7fcf4c0122d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:15:16.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:16.143+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c009800 msgr2=0x7fcf4c0122d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:15:16.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:16.143+0000 7fcf6c8af700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c0127c0 legacy=0x7fcf4c004a10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:15:16.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:16.143+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c009800 0x7fcf4c0122d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:15:17.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:17.740+0000 7fcf5effd700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c009800 msgr2=0x7fcf4c0122d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:15:17.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:17.740+0000 7fcf5effd700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c009800 0x7fcf4c0122d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:15:17.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:17.740+0000 7fcf5effd700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c0127c0 legacy=0x7fcf4c004a10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:15:17.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:17.740+0000 7fcf6d8b1700 1 -- reap_dead start 2023-10-18T16:15:17.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:17.740+0000 7fcf5effd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c00a270 0x7fcf4c009d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:15:17.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:17.740+0000 7fcf5effd700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c005830 0x7fcf4c00f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:15:17.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:17.740+0000 7fcf5effd700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf4c0084e0 con 0x7fcf4c012bd0 2023-10-18T16:15:17.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:17.740+0000 7fcf5effd700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf6814d2b0 con 0x7fcf4c005830 2023-10-18T16:15:17.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:17.740+0000 7fcf5effd700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf4c010160 con 0x7fcf4c00a270 2023-10-18T16:15:17.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:17.740+0000 7fcf6d0b0700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c012bd0 legacy=0x7fcf4c005120 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:15:17.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:17.740+0000 7fcf6c8af700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c00a270 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:15:17.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:17.740+0000 7fcf6c8af700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c00a270 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:15:17.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:17.740+0000 7fcf6d8b1700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c005830 msgr2=0x7fcf4c00f560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:15:17.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:17.740+0000 7fcf6d8b1700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c005830 0x7fcf4c00f560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:15:17.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:17.941+0000 7fcf6d0b0700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c012bd0 legacy=0x7fcf4c005120 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:15:17.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:17.941+0000 7fcf6d8b1700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c005830 msgr2=0x7fcf4c00f560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:15:17.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:17.941+0000 7fcf6c8af700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c00a270 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:15:17.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:17.941+0000 7fcf6c8af700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c00a270 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:15:17.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:17.941+0000 7fcf6d8b1700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c005830 0x7fcf4c00f560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:15:18.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:18.342+0000 7fcf6d8b1700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c005830 msgr2=0x7fcf4c00f560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:15:18.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:18.342+0000 7fcf6d0b0700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c012bd0 legacy=0x7fcf4c005120 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:15:18.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:18.342+0000 7fcf6c8af700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c00a270 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:15:18.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:18.342+0000 7fcf6d8b1700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c005830 0x7fcf4c00f560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:15:18.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:18.342+0000 7fcf6c8af700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c00a270 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:15:19.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:19.143+0000 7fcf6d0b0700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c012bd0 legacy=0x7fcf4c005120 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:15:19.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:19.143+0000 7fcf6c8af700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c00a270 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:15:19.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:19.143+0000 7fcf6d8b1700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c005830 msgr2=0x7fcf4c00f560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:15:19.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:19.143+0000 7fcf6c8af700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c00a270 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:15:19.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:19.143+0000 7fcf6d8b1700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c005830 0x7fcf4c00f560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:15:20.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:20.740+0000 7fcf5effd700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c00a270 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:15:20.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:20.740+0000 7fcf5effd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c00a270 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:15:20.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:20.740+0000 7fcf5effd700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c005830 msgr2=0x7fcf4c00f560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:15:20.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:20.740+0000 7fcf5effd700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c005830 0x7fcf4c00f560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:15:20.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:20.740+0000 7fcf5effd700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c012bd0 legacy=0x7fcf4c005120 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:15:20.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:20.740+0000 7fcf5effd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008c00 0x7fcf4c0122b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:15:20.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:20.740+0000 7fcf5effd700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf680be360 con 0x7fcf4c008fe0 2023-10-18T16:15:20.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:20.740+0000 7fcf5effd700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf6814d2b0 con 0x7fcf4c008c00 2023-10-18T16:15:20.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:20.740+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008c00 msgr2=0x7fcf4c0122b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:15:20.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:20.740+0000 7fcf6c8af700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c008fe0 legacy=0x7fcf4c0093f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:15:20.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:20.740+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008c00 0x7fcf4c0122b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:15:20.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:20.941+0000 7fcf6c8af700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c008fe0 legacy=0x7fcf4c0093f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:15:20.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:20.941+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008c00 msgr2=0x7fcf4c0122b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:15:20.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:20.941+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008c00 0x7fcf4c0122b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:15:21.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:21.342+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008c00 msgr2=0x7fcf4c0122b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:15:21.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:21.342+0000 7fcf6c8af700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c008fe0 legacy=0x7fcf4c0093f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:15:21.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:21.342+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008c00 0x7fcf4c0122b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:15:22.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:22.143+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008c00 msgr2=0x7fcf4c0122b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:15:22.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:22.143+0000 7fcf6c8af700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c008fe0 legacy=0x7fcf4c0093f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:15:22.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:22.143+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008c00 0x7fcf4c0122b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:15:23.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:23.740+0000 7fcf5effd700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008c00 msgr2=0x7fcf4c0122b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:15:23.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:23.740+0000 7fcf5effd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008c00 0x7fcf4c0122b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:15:23.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:23.740+0000 7fcf5effd700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c008fe0 legacy=0x7fcf4c0093f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:15:23.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:23.740+0000 7fcf6d8b1700 1 -- reap_dead start 2023-10-18T16:15:23.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:23.740+0000 7fcf5effd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008c00 0x7fcf4c0122b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:15:23.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:23.740+0000 7fcf5effd700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008fe0 0x7fcf4c009d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:15:23.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:23.740+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008c00 msgr2=0x7fcf4c0122b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:15:23.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:23.740+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008c00 0x7fcf4c0122b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:15:23.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:23.740+0000 7fcf5effd700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf4c0084e0 con 0x7fcf4c00a270 2023-10-18T16:15:23.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:23.740+0000 7fcf5effd700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf6814d2b0 con 0x7fcf4c008c00 2023-10-18T16:15:23.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:23.740+0000 7fcf5effd700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf4c010160 con 0x7fcf4c008fe0 2023-10-18T16:15:23.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:23.740+0000 7fcf6d8b1700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c00a270 legacy=0x7fcf4c0093c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:15:23.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:23.740+0000 7fcf6c8af700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008fe0 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:15:23.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:23.740+0000 7fcf6c8af700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008fe0 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:15:23.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:23.941+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008c00 msgr2=0x7fcf4c0122b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:15:23.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:23.941+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008c00 0x7fcf4c0122b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:15:23.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:23.941+0000 7fcf6d8b1700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c00a270 legacy=0x7fcf4c0093c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:15:23.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:23.941+0000 7fcf6c8af700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008fe0 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:15:23.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:23.941+0000 7fcf6c8af700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008fe0 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:15:24.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:24.342+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008c00 msgr2=0x7fcf4c0122b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:15:24.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:24.342+0000 7fcf6d8b1700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c00a270 legacy=0x7fcf4c0093c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:15:24.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:24.342+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008c00 0x7fcf4c0122b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:15:24.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:24.342+0000 7fcf6c8af700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008fe0 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:15:24.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:24.342+0000 7fcf6c8af700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008fe0 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:15:25.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:25.143+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008c00 msgr2=0x7fcf4c0122b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:15:25.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:25.143+0000 7fcf6d8b1700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c00a270 legacy=0x7fcf4c0093c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:15:25.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:25.143+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008c00 0x7fcf4c0122b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:15:25.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:25.143+0000 7fcf6c8af700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008fe0 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:15:25.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:25.143+0000 7fcf6c8af700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008fe0 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:15:26.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:26.740+0000 7fcf5effd700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008fe0 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:15:26.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:26.740+0000 7fcf5effd700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008fe0 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:15:26.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:26.740+0000 7fcf5effd700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008c00 msgr2=0x7fcf4c0122b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:15:26.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:26.740+0000 7fcf5effd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008c00 0x7fcf4c0122b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:15:26.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:26.740+0000 7fcf5effd700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c00a270 legacy=0x7fcf4c0093c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:15:26.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:26.741+0000 7fcf5effd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c012bd0 0x7fcf4c00f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:15:26.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:26.741+0000 7fcf5effd700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf680be360 con 0x7fcf4c00fa50 2023-10-18T16:15:26.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:26.741+0000 7fcf5effd700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf6814d2b0 con 0x7fcf4c012bd0 2023-10-18T16:15:26.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:26.741+0000 7fcf6c8af700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c00fa50 legacy=0x7fcf4c004a10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:15:26.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:26.741+0000 7fcf6d0b0700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c012bd0 msgr2=0x7fcf4c00f560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:15:26.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:26.741+0000 7fcf6d0b0700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c012bd0 0x7fcf4c00f560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:15:26.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:26.942+0000 7fcf6c8af700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c00fa50 legacy=0x7fcf4c004a10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:15:26.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:26.942+0000 7fcf6d0b0700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c012bd0 msgr2=0x7fcf4c00f560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:15:26.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:26.942+0000 7fcf6d0b0700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c012bd0 0x7fcf4c00f560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:15:27.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:27.343+0000 7fcf6d0b0700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c012bd0 msgr2=0x7fcf4c00f560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:15:27.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:27.343+0000 7fcf6c8af700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c00fa50 legacy=0x7fcf4c004a10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:15:27.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:27.343+0000 7fcf6d0b0700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c012bd0 0x7fcf4c00f560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:15:28.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:28.144+0000 7fcf6d0b0700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c012bd0 msgr2=0x7fcf4c00f560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:15:28.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:28.144+0000 7fcf6c8af700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c00fa50 legacy=0x7fcf4c004a10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:15:28.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:28.144+0000 7fcf6d0b0700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c012bd0 0x7fcf4c00f560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:15:29.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:29.741+0000 7fcf5effd700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c012bd0 msgr2=0x7fcf4c00f560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:15:29.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:29.741+0000 7fcf5effd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c012bd0 0x7fcf4c00f560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:15:29.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:29.741+0000 7fcf5effd700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c00fa50 legacy=0x7fcf4c004a10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:15:29.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:29.741+0000 7fcf6d8b1700 1 -- reap_dead start 2023-10-18T16:15:29.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:29.741+0000 7fcf5effd700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008fe0 0x7fcf4c009d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:15:29.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:29.741+0000 7fcf5effd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c008c00 0x7fcf4c012bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:15:29.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:29.741+0000 7fcf5effd700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf4c0084e0 con 0x7fcf4c00a270 2023-10-18T16:15:29.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:29.741+0000 7fcf5effd700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf6814d2b0 con 0x7fcf4c008fe0 2023-10-18T16:15:29.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:29.741+0000 7fcf5effd700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf4c010160 con 0x7fcf4c008c00 2023-10-18T16:15:29.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:29.741+0000 7fcf6d8b1700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c00a270 legacy=0x7fcf4c0093c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:15:29.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:29.741+0000 7fcf6c8af700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c008c00 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:15:29.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:29.741+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008fe0 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:15:29.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:29.741+0000 7fcf6c8af700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c008c00 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:15:29.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:29.741+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008fe0 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:15:29.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:29.942+0000 7fcf6d8b1700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c00a270 legacy=0x7fcf4c0093c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:15:29.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:29.942+0000 7fcf6c8af700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c008c00 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:15:29.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:29.942+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008fe0 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:15:29.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:29.942+0000 7fcf6c8af700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c008c00 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:15:29.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:29.942+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008fe0 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:15:30.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:30.343+0000 7fcf6d8b1700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c00a270 legacy=0x7fcf4c0093c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:15:30.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:30.343+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008fe0 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:15:30.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:30.343+0000 7fcf6c8af700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c008c00 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:15:30.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:30.343+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008fe0 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:15:30.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:30.343+0000 7fcf6c8af700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c008c00 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:15:31.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:31.144+0000 7fcf6d8b1700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c00a270 legacy=0x7fcf4c0093c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:15:31.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:31.144+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008fe0 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:15:31.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:31.144+0000 7fcf6c8af700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c008c00 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:15:31.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:31.144+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008fe0 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:15:31.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:31.144+0000 7fcf6c8af700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c008c00 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:15:32.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:32.741+0000 7fcf5effd700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c008c00 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:15:32.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:32.741+0000 7fcf5effd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c008c00 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:15:32.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:32.741+0000 7fcf5effd700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008fe0 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:15:32.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:32.741+0000 7fcf5effd700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008fe0 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:15:32.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:32.741+0000 7fcf5effd700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c00a270 legacy=0x7fcf4c0093c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:15:32.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:32.741+0000 7fcf5effd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c005990 0x7fcf4c017710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:15:32.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:32.741+0000 7fcf5effd700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf680be360 con 0x7fcf4c0122b0 2023-10-18T16:15:32.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:32.741+0000 7fcf5effd700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf6814d2b0 con 0x7fcf4c005990 2023-10-18T16:15:32.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:32.741+0000 7fcf6c8af700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c005990 msgr2=0x7fcf4c017710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:15:32.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:32.741+0000 7fcf6c8af700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c005990 0x7fcf4c017710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:15:32.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:32.741+0000 7fcf6d0b0700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c0122b0 legacy=0x7fcf4c00f560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:15:32.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:32.942+0000 7fcf6c8af700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c005990 msgr2=0x7fcf4c017710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:15:32.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:32.942+0000 7fcf6c8af700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c005990 0x7fcf4c017710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:15:32.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:32.942+0000 7fcf6d0b0700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c0122b0 legacy=0x7fcf4c00f560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:15:33.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:33.343+0000 7fcf6c8af700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c005990 msgr2=0x7fcf4c017710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:15:33.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:33.343+0000 7fcf6c8af700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c005990 0x7fcf4c017710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:15:33.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:33.343+0000 7fcf6d0b0700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c0122b0 legacy=0x7fcf4c00f560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:15:34.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:34.144+0000 7fcf6c8af700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c005990 msgr2=0x7fcf4c017710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:15:34.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:34.144+0000 7fcf6c8af700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c005990 0x7fcf4c017710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:15:34.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:34.144+0000 7fcf6d0b0700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c0122b0 legacy=0x7fcf4c00f560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:15:35.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:35.741+0000 7fcf5effd700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c005990 msgr2=0x7fcf4c017710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:15:35.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:35.741+0000 7fcf5effd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c005990 0x7fcf4c017710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:15:35.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:35.741+0000 7fcf5effd700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c0122b0 legacy=0x7fcf4c00f560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:15:35.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:35.741+0000 7fcf6d8b1700 1 -- reap_dead start 2023-10-18T16:15:35.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:35.741+0000 7fcf5effd700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008fe0 0x7fcf4c009d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:15:35.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:35.741+0000 7fcf5effd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008be0 0x7fcf4c012bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:15:35.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:35.741+0000 7fcf5effd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c00a220 0x7fcf4c0093c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:15:35.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:35.741+0000 7fcf5effd700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf6814d2b0 con 0x7fcf4c008be0 2023-10-18T16:15:35.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:35.741+0000 7fcf5effd700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf4c010160 con 0x7fcf4c008fe0 2023-10-18T16:15:35.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:35.741+0000 7fcf5effd700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf4c00a760 con 0x7fcf4c00a220 2023-10-18T16:15:35.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:35.741+0000 7fcf6c8af700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008be0 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:15:35.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:35.741+0000 7fcf6c8af700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008be0 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:15:35.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:35.742+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008fe0 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:15:35.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:35.742+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008fe0 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:15:35.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:35.742+0000 7fcf6d8b1700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c00a220 msgr2=0x7fcf4c0093c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:15:35.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:35.742+0000 7fcf6d8b1700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c00a220 0x7fcf4c0093c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:15:35.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:35.942+0000 7fcf6c8af700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008be0 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:15:35.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:35.942+0000 7fcf6c8af700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008be0 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:15:35.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:35.943+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008fe0 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:15:35.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:35.943+0000 7fcf6d8b1700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c00a220 msgr2=0x7fcf4c0093c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:15:35.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:35.943+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008fe0 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:15:35.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:35.943+0000 7fcf6d8b1700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c00a220 0x7fcf4c0093c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:15:36.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:36.343+0000 7fcf6c8af700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008be0 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:15:36.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:36.343+0000 7fcf6c8af700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008be0 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:15:36.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:36.344+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008fe0 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:15:36.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:36.344+0000 7fcf6d8b1700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c00a220 msgr2=0x7fcf4c0093c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:15:36.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:36.344+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008fe0 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:15:36.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:36.344+0000 7fcf6d8b1700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c00a220 0x7fcf4c0093c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:15:37.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:37.144+0000 7fcf6c8af700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008be0 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:15:37.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:37.144+0000 7fcf6c8af700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008be0 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:15:37.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:37.145+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008fe0 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:15:37.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:37.145+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008fe0 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:15:37.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:37.145+0000 7fcf6d8b1700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c00a220 msgr2=0x7fcf4c0093c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:15:37.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:37.145+0000 7fcf6d8b1700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c00a220 0x7fcf4c0093c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:15:38.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:38.742+0000 7fcf5effd700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c00a220 msgr2=0x7fcf4c0093c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:15:38.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:38.742+0000 7fcf5effd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c00a220 0x7fcf4c0093c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:15:38.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:38.742+0000 7fcf5effd700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008fe0 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:15:38.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:38.742+0000 7fcf5effd700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008fe0 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:15:38.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:38.742+0000 7fcf5effd700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008be0 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:15:38.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:38.742+0000 7fcf5effd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008be0 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:15:38.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:38.742+0000 7fcf5effd700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf4c0084e0 con 0x7fcf4c005980 2023-10-18T16:15:38.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:38.742+0000 7fcf5effd700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf680be360 con 0x7fcf4c0122b0 2023-10-18T16:15:38.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:38.742+0000 7fcf6c8af700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c005980 legacy=0x7fcf4c016710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:15:38.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:38.742+0000 7fcf6d0b0700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c0122b0 legacy=0x7fcf4c00f560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:15:38.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:38.943+0000 7fcf6c8af700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c005980 legacy=0x7fcf4c016710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:15:38.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:38.943+0000 7fcf6d0b0700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c0122b0 legacy=0x7fcf4c00f560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:15:39.305 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-18T16:15:39.308 DEBUG:teuthology.orchestra.run.smithi151:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-18T16:15:39.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:39.344+0000 7fcf6c8af700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c005980 legacy=0x7fcf4c016710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:15:39.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:39.344+0000 7fcf6d0b0700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c0122b0 legacy=0x7fcf4c00f560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:15:40.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:40.145+0000 7fcf6c8af700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c005980 legacy=0x7fcf4c016710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:15:40.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:40.145+0000 7fcf6d0b0700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c0122b0 legacy=0x7fcf4c00f560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:15:41.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:41.742+0000 7fcf5effd700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c0122b0 legacy=0x7fcf4c00f560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:15:41.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:41.742+0000 7fcf5effd700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c005980 legacy=0x7fcf4c016710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:15:41.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:41.742+0000 7fcf6d8b1700 1 -- reap_dead start 2023-10-18T16:15:41.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:41.742+0000 7fcf5effd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008be0 0x7fcf4c012bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:15:41.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:41.742+0000 7fcf5effd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c00f560 0x7fcf4c009d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:15:41.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:41.742+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008be0 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:15:41.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:41.742+0000 7fcf5effd700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c00a220 0x7fcf4c008fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:15:41.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:41.742+0000 7fcf5effd700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf6814d2b0 con 0x7fcf4c008be0 2023-10-18T16:15:41.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:41.742+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008be0 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:15:41.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:41.742+0000 7fcf5effd700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf4c010160 con 0x7fcf4c00a220 2023-10-18T16:15:41.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:41.742+0000 7fcf5effd700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf4c00a760 con 0x7fcf4c00f560 2023-10-18T16:15:41.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:41.742+0000 7fcf6d8b1700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c00a220 msgr2=0x7fcf4c008fc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:15:41.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:41.742+0000 7fcf6c8af700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c00f560 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:15:41.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:41.742+0000 7fcf6c8af700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c00f560 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:15:41.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:41.742+0000 7fcf6d8b1700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c00a220 0x7fcf4c008fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:15:41.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:41.943+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008be0 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:15:41.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:41.943+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008be0 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:15:41.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:41.943+0000 7fcf6d8b1700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c00a220 msgr2=0x7fcf4c008fc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:15:41.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:41.943+0000 7fcf6c8af700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c00f560 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:15:41.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:41.943+0000 7fcf6c8af700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c00f560 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:15:41.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:41.943+0000 7fcf6d8b1700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c00a220 0x7fcf4c008fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:15:42.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:42.344+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008be0 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:15:42.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:42.344+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008be0 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:15:42.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:42.344+0000 7fcf6d8b1700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c00a220 msgr2=0x7fcf4c008fc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:15:42.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:42.344+0000 7fcf6c8af700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c00f560 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:15:42.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:42.344+0000 7fcf6c8af700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c00f560 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:15:42.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:42.344+0000 7fcf6d8b1700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c00a220 0x7fcf4c008fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:15:43.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:43.145+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008be0 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:15:43.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:43.145+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008be0 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:15:43.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:43.145+0000 7fcf6c8af700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c00f560 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:15:43.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:43.145+0000 7fcf6d8b1700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c00a220 msgr2=0x7fcf4c008fc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:15:43.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:43.145+0000 7fcf6c8af700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c00f560 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:15:43.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:43.145+0000 7fcf6d8b1700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c00a220 0x7fcf4c008fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:15:44.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:44.742+0000 7fcf5effd700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c00f560 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:15:44.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:44.742+0000 7fcf5effd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c00f560 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:15:44.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:44.742+0000 7fcf5effd700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c00a220 msgr2=0x7fcf4c008fc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:15:44.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:44.742+0000 7fcf5effd700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c00a220 0x7fcf4c008fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:15:44.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:44.742+0000 7fcf5effd700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008be0 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:15:44.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:44.742+0000 7fcf5effd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008be0 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:15:44.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:44.742+0000 7fcf5effd700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf680be640 con 0x7fcf4c005980 2023-10-18T16:15:44.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:44.742+0000 7fcf5effd700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf4c0084e0 con 0x7fcf4c0094b0 2023-10-18T16:15:44.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:44.742+0000 7fcf6c8af700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c005980 legacy=0x7fcf4c016e20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:15:44.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:44.742+0000 7fcf6d0b0700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c0094b0 legacy=0x7fcf4c0136f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:15:44.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:44.943+0000 7fcf6c8af700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c005980 legacy=0x7fcf4c016e20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:15:44.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:44.943+0000 7fcf6d0b0700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c0094b0 legacy=0x7fcf4c0136f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:15:45.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:45.344+0000 7fcf6c8af700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c005980 legacy=0x7fcf4c016e20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:15:45.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:45.344+0000 7fcf6d0b0700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c0094b0 legacy=0x7fcf4c0136f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:15:46.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:46.145+0000 7fcf6c8af700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c005980 legacy=0x7fcf4c016e20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:15:46.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:46.145+0000 7fcf6d0b0700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c0094b0 legacy=0x7fcf4c0136f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:15:47.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:47.743+0000 7fcf5effd700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c0094b0 legacy=0x7fcf4c0136f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:15:47.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:47.743+0000 7fcf5effd700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c005980 legacy=0x7fcf4c016e20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:15:47.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:47.743+0000 7fcf6d8b1700 1 -- reap_dead start 2023-10-18T16:15:47.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:47.743+0000 7fcf5effd700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c00f560 0x7fcf4c012bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:15:47.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:47.743+0000 7fcf6d0b0700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c0122b0 legacy=0x7fcf4c01a550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:15:47.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:47.743+0000 7fcf5effd700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf680be360 con 0x7fcf4c0122b0 2023-10-18T16:15:47.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:47.743+0000 7fcf5effd700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf680be640 con 0x7fcf4c00f970 2023-10-18T16:15:47.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:47.743+0000 7fcf5effd700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf6814d2b0 con 0x7fcf4c00f560 2023-10-18T16:15:47.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:47.743+0000 7fcf6c8af700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c00f560 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:15:47.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:47.743+0000 7fcf6d8b1700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c00f970 legacy=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:15:47.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:47.743+0000 7fcf6c8af700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c00f560 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:15:47.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:47.944+0000 7fcf6d0b0700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c0122b0 legacy=0x7fcf4c01a550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:15:47.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:47.944+0000 7fcf6c8af700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c00f560 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:15:47.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:47.944+0000 7fcf6d8b1700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c00f970 legacy=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:15:47.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:47.944+0000 7fcf6c8af700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c00f560 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:15:48.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:48.345+0000 7fcf6d0b0700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c0122b0 legacy=0x7fcf4c01a550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:15:48.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:48.345+0000 7fcf6d8b1700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c00f970 legacy=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:15:48.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:48.345+0000 7fcf6c8af700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c00f560 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:15:48.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:48.345+0000 7fcf6c8af700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c00f560 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:15:49.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:49.146+0000 7fcf6d0b0700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c0122b0 legacy=0x7fcf4c01a550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:15:49.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:49.146+0000 7fcf6d8b1700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c00f970 legacy=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:15:49.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:49.146+0000 7fcf6c8af700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c00f560 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:15:49.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:49.146+0000 7fcf6c8af700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c00f560 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:15:50.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:50.743+0000 7fcf5effd700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c00f560 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:15:50.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:50.743+0000 7fcf5effd700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c00f560 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:15:50.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:50.743+0000 7fcf5effd700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c00f970 legacy=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:15:50.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:50.743+0000 7fcf5effd700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c0122b0 legacy=0x7fcf4c01a550 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:15:50.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:50.743+0000 7fcf5effd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c008be0 0x7fcf4c008fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:15:50.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:50.743+0000 7fcf5effd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c0094b0 0x7fcf4c004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:15:50.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:50.743+0000 7fcf5effd700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf6814d2b0 con 0x7fcf4c0094b0 2023-10-18T16:15:50.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:50.743+0000 7fcf5effd700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf4c010160 con 0x7fcf4c008be0 2023-10-18T16:15:50.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:50.743+0000 7fcf6c8af700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c0094b0 msgr2=0x7fcf4c004980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:15:50.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:50.743+0000 7fcf6c8af700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c0094b0 0x7fcf4c004980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:15:50.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:50.743+0000 7fcf6d0b0700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c008be0 msgr2=0x7fcf4c008fc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:15:50.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:50.743+0000 7fcf6d0b0700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c008be0 0x7fcf4c008fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:15:50.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:50.944+0000 7fcf6c8af700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c0094b0 msgr2=0x7fcf4c004980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:15:50.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:50.944+0000 7fcf6c8af700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c0094b0 0x7fcf4c004980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:15:50.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:50.944+0000 7fcf6d0b0700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c008be0 msgr2=0x7fcf4c008fc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:15:50.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:50.944+0000 7fcf6d0b0700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c008be0 0x7fcf4c008fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:15:51.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:51.345+0000 7fcf6c8af700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c0094b0 msgr2=0x7fcf4c004980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:15:51.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:51.345+0000 7fcf6c8af700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c0094b0 0x7fcf4c004980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:15:51.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:51.345+0000 7fcf6d0b0700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c008be0 msgr2=0x7fcf4c008fc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:15:51.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:51.345+0000 7fcf6d0b0700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c008be0 0x7fcf4c008fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:15:52.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:52.146+0000 7fcf6c8af700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c0094b0 msgr2=0x7fcf4c004980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:15:52.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:52.146+0000 7fcf6c8af700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c0094b0 0x7fcf4c004980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:15:52.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:52.146+0000 7fcf6d0b0700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c008be0 msgr2=0x7fcf4c008fc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:15:52.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:52.146+0000 7fcf6d0b0700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c008be0 0x7fcf4c008fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:15:53.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:53.744+0000 7fcf5effd700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c008be0 msgr2=0x7fcf4c008fc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:15:53.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:53.744+0000 7fcf5effd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c008be0 0x7fcf4c008fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:15:53.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:53.744+0000 7fcf5effd700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c0094b0 msgr2=0x7fcf4c004980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:15:53.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:53.744+0000 7fcf5effd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c0094b0 0x7fcf4c004980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:15:53.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:53.744+0000 7fcf6d8b1700 1 -- reap_dead start 2023-10-18T16:15:53.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:53.744+0000 7fcf5effd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c005050 0x7fcf4c005430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:15:53.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:53.744+0000 7fcf5effd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c008be0 0x7fcf4c004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:15:53.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:53.744+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c005050 msgr2=0x7fcf4c005430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:15:53.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:53.744+0000 7fcf5effd700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf4c0084e0 con 0x7fcf4c00f560 2023-10-18T16:15:53.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:53.744+0000 7fcf5effd700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf6814d2b0 con 0x7fcf4c005050 2023-10-18T16:15:53.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:53.744+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c005050 0x7fcf4c005430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:15:53.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:53.744+0000 7fcf5effd700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf4c010160 con 0x7fcf4c008be0 2023-10-18T16:15:53.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:53.744+0000 7fcf6c8af700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c008be0 msgr2=0x7fcf4c004980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:15:53.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:53.744+0000 7fcf6d8b1700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c00f560 legacy=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:15:53.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:53.744+0000 7fcf6c8af700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c008be0 0x7fcf4c004980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:15:53.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:53.945+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c005050 msgr2=0x7fcf4c005430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:15:53.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:53.945+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c005050 0x7fcf4c005430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:15:53.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:53.945+0000 7fcf6c8af700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c008be0 msgr2=0x7fcf4c004980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:15:53.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:53.945+0000 7fcf6d8b1700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c00f560 legacy=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:15:53.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:53.945+0000 7fcf6c8af700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c008be0 0x7fcf4c004980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:15:54.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:54.346+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c005050 msgr2=0x7fcf4c005430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:15:54.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:54.346+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c005050 0x7fcf4c005430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:15:54.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:54.346+0000 7fcf6c8af700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c008be0 msgr2=0x7fcf4c004980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:15:54.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:54.346+0000 7fcf6d8b1700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c00f560 legacy=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:15:54.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:54.346+0000 7fcf6c8af700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c008be0 0x7fcf4c004980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:15:55.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:55.147+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c005050 msgr2=0x7fcf4c005430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:15:55.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:55.147+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c005050 0x7fcf4c005430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:15:55.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:55.147+0000 7fcf6c8af700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c008be0 msgr2=0x7fcf4c004980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:15:55.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:55.147+0000 7fcf6d8b1700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c00f560 legacy=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:15:55.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:55.147+0000 7fcf6c8af700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c008be0 0x7fcf4c004980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:15:56.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:56.743+0000 7fcf5effd700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c008be0 msgr2=0x7fcf4c004980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:15:56.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:56.743+0000 7fcf5effd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c008be0 0x7fcf4c004980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:15:56.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:56.743+0000 7fcf5effd700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c005050 msgr2=0x7fcf4c005430 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:15:56.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:56.743+0000 7fcf5effd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c005050 0x7fcf4c005430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:15:56.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:56.743+0000 7fcf5effd700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c00f560 legacy=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:15:56.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:56.743+0000 7fcf5effd700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008fc0 0x7fcf4c012bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:15:56.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:56.743+0000 7fcf5effd700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf680be360 con 0x7fcf4c00f970 2023-10-18T16:15:56.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:56.743+0000 7fcf5effd700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf6814d2b0 con 0x7fcf4c008fc0 2023-10-18T16:15:56.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:56.743+0000 7fcf6d0b0700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c00f970 legacy=0x7fcf4c0102a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:15:56.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:56.743+0000 7fcf6c8af700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008fc0 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:15:56.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:56.743+0000 7fcf6c8af700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008fc0 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:15:56.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:56.944+0000 7fcf6d0b0700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c00f970 legacy=0x7fcf4c0102a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:15:56.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:56.944+0000 7fcf6c8af700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008fc0 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:15:56.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:56.944+0000 7fcf6c8af700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008fc0 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:15:57.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:57.345+0000 7fcf6d0b0700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c00f970 legacy=0x7fcf4c0102a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:15:57.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:57.345+0000 7fcf6c8af700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008fc0 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:15:57.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:57.345+0000 7fcf6c8af700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008fc0 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:15:58.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:58.146+0000 7fcf6d0b0700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c00f970 legacy=0x7fcf4c0102a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:15:58.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:58.146+0000 7fcf6c8af700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008fc0 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:15:58.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:58.146+0000 7fcf6c8af700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008fc0 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:15:59.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:59.744+0000 7fcf5effd700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008fc0 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:15:59.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:59.744+0000 7fcf5effd700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008fc0 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:15:59.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:59.744+0000 7fcf5effd700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c00f970 legacy=0x7fcf4c0102a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:15:59.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:59.744+0000 7fcf6d8b1700 1 -- reap_dead start 2023-10-18T16:15:59.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:59.744+0000 7fcf5effd700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008be0 0x7fcf4c012bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:15:59.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:59.744+0000 7fcf5effd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c00f560 0x7fcf4c009d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:15:59.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:59.744+0000 7fcf5effd700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf4c0084e0 con 0x7fcf4c00f970 2023-10-18T16:15:59.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:59.744+0000 7fcf5effd700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf6814d2b0 con 0x7fcf4c00f560 2023-10-18T16:15:59.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:59.744+0000 7fcf5effd700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf4c010160 con 0x7fcf4c008be0 2023-10-18T16:15:59.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:59.744+0000 7fcf6c8af700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c00f560 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:15:59.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:59.744+0000 7fcf6c8af700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c00f560 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:15:59.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:59.744+0000 7fcf6d8b1700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c00f970 legacy=0x7fcf4c005050 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:15:59.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:59.744+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008be0 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:15:59.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:59.744+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008be0 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:15:59.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:59.945+0000 7fcf6c8af700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c00f560 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:15:59.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:59.945+0000 7fcf6d8b1700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c00f970 legacy=0x7fcf4c005050 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:15:59.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:59.945+0000 7fcf6c8af700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c00f560 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:15:59.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:59.945+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008be0 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:15:59.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:15:59.945+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008be0 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:16:00.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:00.346+0000 7fcf6d8b1700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c00f970 legacy=0x7fcf4c005050 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:16:00.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:00.346+0000 7fcf6c8af700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c00f560 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:16:00.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:00.346+0000 7fcf6c8af700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c00f560 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:16:00.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:00.346+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008be0 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:16:00.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:00.346+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008be0 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:16:01.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:01.147+0000 7fcf6d8b1700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c00f970 legacy=0x7fcf4c005050 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:16:01.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:01.147+0000 7fcf6c8af700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c00f560 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:16:01.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:01.147+0000 7fcf6c8af700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c00f560 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:16:01.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:01.147+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008be0 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:16:01.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:01.147+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008be0 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:16:02.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:02.745+0000 7fcf5effd700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008be0 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:16:02.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:02.745+0000 7fcf5effd700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008be0 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:16:02.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:02.745+0000 7fcf5effd700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c00f560 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:16:02.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:02.745+0000 7fcf5effd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c00f560 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:16:02.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:02.745+0000 7fcf5effd700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c00f970 legacy=0x7fcf4c005050 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:16:02.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:02.745+0000 7fcf5effd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c004980 0x7fcf4c018e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:16:02.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:02.745+0000 7fcf5effd700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf680be360 con 0x7fcf4c0122b0 2023-10-18T16:16:02.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:02.745+0000 7fcf5effd700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf6814d2b0 con 0x7fcf4c004980 2023-10-18T16:16:02.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:02.745+0000 7fcf6c8af700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c004980 msgr2=0x7fcf4c018e30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:16:02.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:02.745+0000 7fcf6d0b0700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c0122b0 legacy=0x7fcf4c015700 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:16:02.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:02.745+0000 7fcf6c8af700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c004980 0x7fcf4c018e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:16:02.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:02.946+0000 7fcf6d0b0700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c0122b0 legacy=0x7fcf4c015700 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:16:02.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:02.946+0000 7fcf6c8af700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c004980 msgr2=0x7fcf4c018e30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:16:02.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:02.946+0000 7fcf6c8af700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c004980 0x7fcf4c018e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:16:03.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:03.347+0000 7fcf6d0b0700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c0122b0 legacy=0x7fcf4c015700 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:16:03.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:03.347+0000 7fcf6c8af700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c004980 msgr2=0x7fcf4c018e30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:16:03.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:03.347+0000 7fcf6c8af700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c004980 0x7fcf4c018e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:16:04.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:04.148+0000 7fcf6c8af700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c004980 msgr2=0x7fcf4c018e30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:16:04.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:04.148+0000 7fcf6d0b0700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c0122b0 legacy=0x7fcf4c015700 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:16:04.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:04.148+0000 7fcf6c8af700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c004980 0x7fcf4c018e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:16:05.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:05.744+0000 7fcf5effd700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c004980 msgr2=0x7fcf4c018e30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:16:05.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:05.744+0000 7fcf5effd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c004980 0x7fcf4c018e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:16:05.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:05.744+0000 7fcf5effd700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c0122b0 legacy=0x7fcf4c015700 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:16:05.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:05.744+0000 7fcf6d8b1700 1 -- reap_dead start 2023-10-18T16:16:05.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:05.744+0000 7fcf5effd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008be0 0x7fcf4c012bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:16:05.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:05.744+0000 7fcf5effd700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c00f560 0x7fcf4c009d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:16:05.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:05.744+0000 7fcf5effd700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf4c0084e0 con 0x7fcf4c00f970 2023-10-18T16:16:05.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:05.744+0000 7fcf5effd700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf6814d2b0 con 0x7fcf4c008be0 2023-10-18T16:16:05.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:05.744+0000 7fcf5effd700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf4c010160 con 0x7fcf4c00f560 2023-10-18T16:16:05.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:05.744+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008be0 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:16:05.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:05.744+0000 7fcf6d8b1700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c00f970 legacy=0x7fcf4c005020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:16:05.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:05.744+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008be0 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:16:05.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:05.744+0000 7fcf6c8af700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c00f560 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:16:05.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:05.744+0000 7fcf6c8af700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c00f560 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:16:05.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:05.945+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008be0 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:16:05.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:05.945+0000 7fcf6d8b1700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c00f970 legacy=0x7fcf4c005020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:16:05.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:05.945+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008be0 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:16:05.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:05.945+0000 7fcf6c8af700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c00f560 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:16:05.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:05.945+0000 7fcf6c8af700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c00f560 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:16:06.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:06.346+0000 7fcf6d8b1700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c00f970 legacy=0x7fcf4c005020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:16:06.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:06.346+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008be0 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:16:06.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:06.346+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008be0 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:16:06.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:06.346+0000 7fcf6c8af700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c00f560 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:16:06.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:06.346+0000 7fcf6c8af700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c00f560 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:16:07.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:07.147+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008be0 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:16:07.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:07.147+0000 7fcf6d8b1700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c00f970 legacy=0x7fcf4c005020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:16:07.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:07.147+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008be0 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:16:07.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:07.147+0000 7fcf6c8af700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c00f560 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:16:07.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:07.147+0000 7fcf6c8af700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c00f560 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:16:08.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:08.744+0000 7fcf5effd700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c00f560 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:16:08.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:08.744+0000 7fcf5effd700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c00f560 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:16:08.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:08.744+0000 7fcf5effd700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008be0 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:16:08.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:08.744+0000 7fcf5effd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008be0 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:16:08.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:08.744+0000 7fcf5effd700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c00f970 legacy=0x7fcf4c005020 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:16:08.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:08.744+0000 7fcf5effd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c004980 0x7fcf4c0122b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:16:08.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:08.744+0000 7fcf5effd700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf680be360 con 0x7fcf4c015700 2023-10-18T16:16:08.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:08.744+0000 7fcf5effd700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf6814d2b0 con 0x7fcf4c004980 2023-10-18T16:16:08.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:08.744+0000 7fcf6c8af700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c015700 legacy=0x7fcf4c017af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:16:08.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:08.744+0000 7fcf6d0b0700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c004980 msgr2=0x7fcf4c0122b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:16:08.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:08.744+0000 7fcf6d0b0700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c004980 0x7fcf4c0122b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:16:08.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:08.945+0000 7fcf6c8af700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c015700 legacy=0x7fcf4c017af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:16:08.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:08.945+0000 7fcf6d0b0700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c004980 msgr2=0x7fcf4c0122b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:16:08.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:08.945+0000 7fcf6d0b0700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c004980 0x7fcf4c0122b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:16:09.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:09.346+0000 7fcf6c8af700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c015700 legacy=0x7fcf4c017af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:16:09.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:09.346+0000 7fcf6d0b0700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c004980 msgr2=0x7fcf4c0122b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:16:09.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:09.346+0000 7fcf6d0b0700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c004980 0x7fcf4c0122b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:16:09.386 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-18T16:16:09.389 DEBUG:teuthology.orchestra.run.smithi151:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-18T16:16:10.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:10.147+0000 7fcf6d0b0700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c004980 msgr2=0x7fcf4c0122b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:16:10.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:10.147+0000 7fcf6c8af700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c015700 legacy=0x7fcf4c017af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:16:10.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:10.147+0000 7fcf6d0b0700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c004980 0x7fcf4c0122b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:16:11.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:11.744+0000 7fcf5effd700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c004980 msgr2=0x7fcf4c0122b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:16:11.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:11.744+0000 7fcf5effd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c004980 0x7fcf4c0122b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:16:11.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:11.744+0000 7fcf5effd700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c015700 legacy=0x7fcf4c017af0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:16:11.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:11.744+0000 7fcf6d8b1700 1 -- reap_dead start 2023-10-18T16:16:11.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:11.744+0000 7fcf5effd700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c00f560 0x7fcf4c012bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:16:11.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:11.744+0000 7fcf5effd700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf4c0084e0 con 0x7fcf4c008be0 2023-10-18T16:16:11.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:11.744+0000 7fcf5effd700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf680be360 con 0x7fcf4c019210 2023-10-18T16:16:11.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:11.744+0000 7fcf5effd700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf6814d2b0 con 0x7fcf4c00f560 2023-10-18T16:16:11.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:11.744+0000 7fcf6c8af700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c008be0 legacy=0x7fcf4c005020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:16:11.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:11.745+0000 7fcf6d0b0700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c019210 legacy=0x7fcf4c01b600 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:16:11.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:11.745+0000 7fcf6d8b1700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c00f560 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:16:11.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:11.745+0000 7fcf6d8b1700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c00f560 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:16:11.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:11.946+0000 7fcf6c8af700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c008be0 legacy=0x7fcf4c005020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:16:11.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:11.946+0000 7fcf6d0b0700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c019210 legacy=0x7fcf4c01b600 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:16:11.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:11.946+0000 7fcf6d8b1700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c00f560 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:16:11.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:11.946+0000 7fcf6d8b1700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c00f560 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:16:12.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:12.347+0000 7fcf6c8af700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c008be0 legacy=0x7fcf4c005020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:16:12.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:12.347+0000 7fcf6d8b1700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c00f560 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:16:12.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:12.347+0000 7fcf6d0b0700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c019210 legacy=0x7fcf4c01b600 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:16:12.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:12.347+0000 7fcf6d8b1700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c00f560 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:16:13.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:13.148+0000 7fcf6c8af700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c008be0 legacy=0x7fcf4c005020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:16:13.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:13.148+0000 7fcf6d8b1700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c00f560 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:16:13.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:13.148+0000 7fcf6d0b0700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c019210 legacy=0x7fcf4c01b600 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:16:13.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:13.148+0000 7fcf6d8b1700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c00f560 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:16:14.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:14.745+0000 7fcf5effd700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c00f560 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:16:14.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:14.745+0000 7fcf5effd700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c00f560 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:16:14.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:14.745+0000 7fcf5effd700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c019210 legacy=0x7fcf4c01b600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:16:14.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:14.745+0000 7fcf5effd700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c008be0 legacy=0x7fcf4c005020 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:16:14.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:14.745+0000 7fcf5effd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c00f970 0x7fcf4c009d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:16:14.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:14.745+0000 7fcf5effd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c004980 0x7fcf4c0122b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:16:14.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:14.745+0000 7fcf5effd700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf6814d2b0 con 0x7fcf4c00f970 2023-10-18T16:16:14.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:14.745+0000 7fcf5effd700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf4c010160 con 0x7fcf4c004980 2023-10-18T16:16:14.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:14.745+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c00f970 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:16:14.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:14.745+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c00f970 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:16:14.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:14.745+0000 7fcf6c8af700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c004980 msgr2=0x7fcf4c0122b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:16:14.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:14.745+0000 7fcf6c8af700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c004980 0x7fcf4c0122b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:16:14.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:14.946+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c00f970 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:16:14.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:14.946+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c00f970 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:16:14.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:14.946+0000 7fcf6c8af700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c004980 msgr2=0x7fcf4c0122b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:16:14.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:14.946+0000 7fcf6c8af700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c004980 0x7fcf4c0122b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:16:15.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:15.347+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c00f970 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:16:15.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:15.347+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c00f970 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:16:15.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:15.347+0000 7fcf6c8af700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c004980 msgr2=0x7fcf4c0122b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:16:15.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:15.347+0000 7fcf6c8af700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c004980 0x7fcf4c0122b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:16:16.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:16.148+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c00f970 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:16:16.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:16.148+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c00f970 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:16:16.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:16.148+0000 7fcf6c8af700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c004980 msgr2=0x7fcf4c0122b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:16:16.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:16.148+0000 7fcf6c8af700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c004980 0x7fcf4c0122b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:16:17.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:17.745+0000 7fcf5effd700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c004980 msgr2=0x7fcf4c0122b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:16:17.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:17.745+0000 7fcf5effd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c004980 0x7fcf4c0122b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:16:17.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:17.745+0000 7fcf5effd700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c00f970 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:16:17.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:17.745+0000 7fcf5effd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c00f970 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:16:17.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:17.745+0000 7fcf5effd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c015ae0 0x7fcf4c017ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:16:17.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:17.745+0000 7fcf5effd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c00f970 0x7fcf4c009d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:16:17.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:17.745+0000 7fcf6d8b1700 1 -- reap_dead start 2023-10-18T16:16:17.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:17.745+0000 7fcf5effd700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf680be640 con 0x7fcf4c008be0 2023-10-18T16:16:17.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:17.745+0000 7fcf5effd700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf6814d2b0 con 0x7fcf4c00f970 2023-10-18T16:16:17.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:17.745+0000 7fcf5effd700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf4c010160 con 0x7fcf4c015ae0 2023-10-18T16:16:17.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:17.745+0000 7fcf6c8af700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c00f970 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:16:17.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:17.745+0000 7fcf6c8af700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c00f970 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:16:17.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:17.745+0000 7fcf6d0b0700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c015ae0 msgr2=0x7fcf4c017ed0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:16:17.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:17.745+0000 7fcf6d8b1700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c008be0 legacy=0x7fcf4c005020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:16:17.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:17.745+0000 7fcf6d0b0700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c015ae0 0x7fcf4c017ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:16:17.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:17.946+0000 7fcf6c8af700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c00f970 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:16:17.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:17.946+0000 7fcf6c8af700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c00f970 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:16:17.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:17.946+0000 7fcf6d8b1700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c008be0 legacy=0x7fcf4c005020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:16:17.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:17.946+0000 7fcf6d0b0700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c015ae0 msgr2=0x7fcf4c017ed0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:16:17.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:17.946+0000 7fcf6d0b0700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c015ae0 0x7fcf4c017ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:16:18.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:18.347+0000 7fcf6c8af700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c00f970 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:16:18.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:18.347+0000 7fcf6c8af700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c00f970 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:16:18.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:18.347+0000 7fcf6d0b0700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c015ae0 msgr2=0x7fcf4c017ed0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:16:18.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:18.347+0000 7fcf6d8b1700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c008be0 legacy=0x7fcf4c005020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:16:18.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:18.347+0000 7fcf6d0b0700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c015ae0 0x7fcf4c017ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:16:19.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:19.148+0000 7fcf6c8af700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c00f970 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:16:19.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:19.148+0000 7fcf6c8af700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c00f970 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:16:19.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:19.148+0000 7fcf6d0b0700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c015ae0 msgr2=0x7fcf4c017ed0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:16:19.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:19.148+0000 7fcf6d0b0700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c015ae0 0x7fcf4c017ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:16:19.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:19.148+0000 7fcf6d8b1700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c008be0 legacy=0x7fcf4c005020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:16:20.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:20.745+0000 7fcf5effd700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c015ae0 msgr2=0x7fcf4c017ed0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:16:20.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:20.745+0000 7fcf5effd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c015ae0 0x7fcf4c017ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:16:20.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:20.745+0000 7fcf5effd700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c00f970 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:16:20.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:20.745+0000 7fcf5effd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c00f970 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:16:20.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:20.745+0000 7fcf5effd700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c008be0 legacy=0x7fcf4c005020 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:16:20.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:20.745+0000 7fcf5effd700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c00f560 0x7fcf4c012bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:16:20.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:20.745+0000 7fcf5effd700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf4c0084e0 con 0x7fcf4c0102a0 2023-10-18T16:16:20.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:20.745+0000 7fcf5effd700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf6814d2b0 con 0x7fcf4c00f560 2023-10-18T16:16:20.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:20.745+0000 7fcf6c8af700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c0102a0 legacy=0x7fcf4c0183c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:16:20.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:20.745+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c00f560 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:16:20.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:20.745+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c00f560 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:16:20.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:20.946+0000 7fcf6c8af700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c0102a0 legacy=0x7fcf4c0183c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:16:20.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:20.946+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c00f560 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:16:20.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:20.946+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c00f560 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:16:21.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:21.347+0000 7fcf6c8af700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c0102a0 legacy=0x7fcf4c0183c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:16:21.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:21.347+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c00f560 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:16:21.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:21.347+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c00f560 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:16:22.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:22.148+0000 7fcf6c8af700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c0102a0 legacy=0x7fcf4c0183c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:16:22.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:22.148+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c00f560 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:16:22.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:22.148+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c00f560 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:16:23.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:23.745+0000 7fcf5effd700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c00f560 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:16:23.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:23.745+0000 7fcf5effd700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c00f560 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:16:23.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:23.745+0000 7fcf5effd700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c0102a0 legacy=0x7fcf4c0183c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:16:23.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:23.745+0000 7fcf6d8b1700 1 -- reap_dead start 2023-10-18T16:16:23.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:23.745+0000 7fcf5effd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c00f970 0x7fcf4c009d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:16:23.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:23.745+0000 7fcf5effd700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008be0 0x7fcf4c012bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:16:23.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:23.745+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c00f970 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:16:23.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:23.745+0000 7fcf5effd700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf680be640 con 0x7fcf4c00f560 2023-10-18T16:16:23.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:23.745+0000 7fcf5effd700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf6814d2b0 con 0x7fcf4c00f970 2023-10-18T16:16:23.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:23.745+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c00f970 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:16:23.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:23.745+0000 7fcf5effd700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf4c010160 con 0x7fcf4c008be0 2023-10-18T16:16:23.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:23.745+0000 7fcf6c8af700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008be0 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:16:23.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:23.745+0000 7fcf6d8b1700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c00f560 legacy=0x7fcf4c005020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:16:23.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:23.745+0000 7fcf6c8af700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008be0 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:16:23.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:23.946+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c00f970 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:16:23.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:23.946+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c00f970 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:16:23.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:23.946+0000 7fcf6d8b1700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c00f560 legacy=0x7fcf4c005020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:16:23.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:23.946+0000 7fcf6c8af700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008be0 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:16:23.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:23.946+0000 7fcf6c8af700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008be0 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:16:24.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:24.347+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c00f970 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:16:24.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:24.347+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c00f970 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:16:24.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:24.347+0000 7fcf6d8b1700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c00f560 legacy=0x7fcf4c005020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:16:24.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:24.347+0000 7fcf6c8af700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008be0 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:16:24.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:24.347+0000 7fcf6c8af700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008be0 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:16:25.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:25.148+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c00f970 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:16:25.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:25.148+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c00f970 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:16:25.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:25.148+0000 7fcf6d8b1700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c00f560 legacy=0x7fcf4c005020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:16:25.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:25.148+0000 7fcf6c8af700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008be0 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:16:25.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:25.148+0000 7fcf6c8af700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008be0 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:16:26.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:26.746+0000 7fcf5effd700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008be0 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:16:26.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:26.746+0000 7fcf5effd700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008be0 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:16:26.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:26.746+0000 7fcf5effd700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c00f970 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:16:26.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:26.746+0000 7fcf5effd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c00f970 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:16:26.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:26.746+0000 7fcf5effd700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c00f560 legacy=0x7fcf4c005020 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:16:26.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:26.746+0000 7fcf5effd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c017200 0x7fcf4c0195f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:16:26.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:26.746+0000 7fcf5effd700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf4c0084e0 con 0x7fcf4c0102a0 2023-10-18T16:16:26.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:26.746+0000 7fcf5effd700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf6814d2b0 con 0x7fcf4c017200 2023-10-18T16:16:26.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:26.746+0000 7fcf6d0b0700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c0102a0 legacy=0x7fcf4c015ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:16:26.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:26.746+0000 7fcf6c8af700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c017200 msgr2=0x7fcf4c0195f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:16:26.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:26.746+0000 7fcf6c8af700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c017200 0x7fcf4c0195f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:16:26.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:26.947+0000 7fcf6d0b0700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c0102a0 legacy=0x7fcf4c015ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:16:26.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:26.947+0000 7fcf6c8af700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c017200 msgr2=0x7fcf4c0195f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:16:26.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:26.947+0000 7fcf6c8af700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c017200 0x7fcf4c0195f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:16:27.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:27.348+0000 7fcf6d0b0700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c0102a0 legacy=0x7fcf4c015ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:16:27.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:27.348+0000 7fcf6c8af700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c017200 msgr2=0x7fcf4c0195f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:16:27.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:27.348+0000 7fcf6c8af700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c017200 0x7fcf4c0195f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:16:28.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:28.149+0000 7fcf6d0b0700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c0102a0 legacy=0x7fcf4c015ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:16:28.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:28.149+0000 7fcf6c8af700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c017200 msgr2=0x7fcf4c0195f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:16:28.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:28.149+0000 7fcf6c8af700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c017200 0x7fcf4c0195f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:16:29.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:29.746+0000 7fcf5effd700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c017200 msgr2=0x7fcf4c0195f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:16:29.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:29.746+0000 7fcf5effd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c017200 0x7fcf4c0195f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:16:29.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:29.746+0000 7fcf5effd700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c0102a0 legacy=0x7fcf4c015ae0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:16:29.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:29.746+0000 7fcf6d8b1700 1 -- reap_dead start 2023-10-18T16:16:29.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:29.746+0000 7fcf5effd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c019ae0 0x7fcf4c01bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:16:29.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:29.746+0000 7fcf5effd700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008be0 0x7fcf4c009d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:16:29.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:29.746+0000 7fcf5effd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c00f560 0x7fcf4c012bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:16:29.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:29.746+0000 7fcf5effd700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf6814d2b0 con 0x7fcf4c00f560 2023-10-18T16:16:29.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:29.746+0000 7fcf5effd700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf4c010160 con 0x7fcf4c008be0 2023-10-18T16:16:29.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:29.746+0000 7fcf5effd700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf4c00a760 con 0x7fcf4c019ae0 2023-10-18T16:16:29.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:29.746+0000 7fcf6d8b1700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c00f560 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:16:29.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:29.746+0000 7fcf6d8b1700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c00f560 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:16:29.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:29.746+0000 7fcf6d0b0700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c019ae0 msgr2=0x7fcf4c01bed0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:16:29.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:29.746+0000 7fcf6c8af700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008be0 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:16:29.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:29.746+0000 7fcf6d0b0700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c019ae0 0x7fcf4c01bed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:16:29.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:29.746+0000 7fcf6c8af700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008be0 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:16:29.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:29.947+0000 7fcf6d8b1700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c00f560 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:16:29.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:29.947+0000 7fcf6d8b1700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c00f560 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:16:29.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:29.947+0000 7fcf6d0b0700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c019ae0 msgr2=0x7fcf4c01bed0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:16:29.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:29.947+0000 7fcf6d0b0700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c019ae0 0x7fcf4c01bed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:16:29.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:29.947+0000 7fcf6c8af700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008be0 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:16:29.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:29.947+0000 7fcf6c8af700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008be0 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:16:30.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:30.348+0000 7fcf6d8b1700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c00f560 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:16:30.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:30.348+0000 7fcf6d8b1700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c00f560 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:16:30.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:30.348+0000 7fcf6d0b0700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c019ae0 msgr2=0x7fcf4c01bed0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:16:30.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:30.348+0000 7fcf6c8af700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008be0 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:16:30.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:30.348+0000 7fcf6d0b0700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c019ae0 0x7fcf4c01bed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:16:30.353 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:30.348+0000 7fcf6c8af700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008be0 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:16:31.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:31.149+0000 7fcf6d8b1700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c00f560 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:16:31.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:31.149+0000 7fcf6d8b1700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c00f560 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:16:31.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:31.149+0000 7fcf6c8af700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008be0 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:16:31.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:31.149+0000 7fcf6d0b0700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c019ae0 msgr2=0x7fcf4c01bed0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:16:31.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:31.149+0000 7fcf6d0b0700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c019ae0 0x7fcf4c01bed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:16:31.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:31.149+0000 7fcf6c8af700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008be0 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:16:32.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:32.746+0000 7fcf5effd700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c019ae0 msgr2=0x7fcf4c01bed0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:16:32.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:32.746+0000 7fcf5effd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c019ae0 0x7fcf4c01bed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:16:32.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:32.746+0000 7fcf5effd700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008be0 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:16:32.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:32.746+0000 7fcf5effd700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c008be0 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:16:32.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:32.746+0000 7fcf5effd700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c00f560 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:16:32.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:32.746+0000 7fcf5effd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c00f560 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:16:32.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:32.746+0000 7fcf5effd700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf680be640 con 0x7fcf4c00f970 2023-10-18T16:16:32.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:32.746+0000 7fcf5effd700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf4c0084e0 con 0x7fcf4c015ae0 2023-10-18T16:16:32.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:32.746+0000 7fcf6d0b0700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c00f970 legacy=0x7fcf4c005020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:16:32.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:32.746+0000 7fcf6c8af700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c015ae0 legacy=0x7fcf4c015ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:16:32.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:32.947+0000 7fcf6d0b0700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c00f970 legacy=0x7fcf4c005020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:16:32.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:32.947+0000 7fcf6c8af700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c015ae0 legacy=0x7fcf4c015ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:16:33.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:33.348+0000 7fcf6d0b0700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c00f970 legacy=0x7fcf4c005020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:16:33.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:33.348+0000 7fcf6c8af700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c015ae0 legacy=0x7fcf4c015ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:16:34.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:34.149+0000 7fcf6d0b0700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c00f970 legacy=0x7fcf4c005020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:16:34.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:34.149+0000 7fcf6c8af700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c015ae0 legacy=0x7fcf4c015ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:16:35.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:35.747+0000 7fcf5effd700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c015ae0 legacy=0x7fcf4c015ef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:16:35.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:35.747+0000 7fcf5effd700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c00f970 legacy=0x7fcf4c005020 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:16:35.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:35.747+0000 7fcf6d8b1700 1 -- reap_dead start 2023-10-18T16:16:35.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:35.747+0000 7fcf5effd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008be0 0x7fcf4c009d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:16:35.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:35.747+0000 7fcf5effd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c00f560 0x7fcf4c016d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:16:35.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:35.747+0000 7fcf5effd700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf680be360 con 0x7fcf4c0122b0 2023-10-18T16:16:35.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:35.747+0000 7fcf5effd700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf6814d2b0 con 0x7fcf4c008be0 2023-10-18T16:16:35.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:35.747+0000 7fcf5effd700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf4c010160 con 0x7fcf4c00f560 2023-10-18T16:16:35.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:35.747+0000 7fcf6c8af700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008be0 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:16:35.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:35.747+0000 7fcf6c8af700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008be0 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:16:35.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:35.747+0000 7fcf6d0b0700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c0122b0 legacy=0x7fcf4c016600 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:16:35.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:35.747+0000 7fcf6d8b1700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c00f560 msgr2=0x7fcf4c016d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:16:35.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:35.747+0000 7fcf6d8b1700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c00f560 0x7fcf4c016d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:16:35.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:35.948+0000 7fcf6c8af700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008be0 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:16:35.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:35.948+0000 7fcf6c8af700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008be0 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:16:35.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:35.948+0000 7fcf6d0b0700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c0122b0 legacy=0x7fcf4c016600 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:16:35.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:35.948+0000 7fcf6d8b1700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c00f560 msgr2=0x7fcf4c016d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:16:35.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:35.948+0000 7fcf6d8b1700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c00f560 0x7fcf4c016d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:16:36.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:36.349+0000 7fcf6c8af700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008be0 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:16:36.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:36.349+0000 7fcf6c8af700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008be0 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:16:36.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:36.349+0000 7fcf6d0b0700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c0122b0 legacy=0x7fcf4c016600 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:16:36.353 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:36.349+0000 7fcf6d8b1700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c00f560 msgr2=0x7fcf4c016d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:16:36.353 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:36.349+0000 7fcf6d8b1700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c00f560 0x7fcf4c016d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:16:37.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:37.150+0000 7fcf6c8af700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008be0 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:16:37.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:37.150+0000 7fcf6c8af700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008be0 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:16:37.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:37.150+0000 7fcf6d8b1700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c00f560 msgr2=0x7fcf4c016d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:16:37.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:37.150+0000 7fcf6d8b1700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c00f560 0x7fcf4c016d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:16:37.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:37.150+0000 7fcf6d0b0700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c0122b0 legacy=0x7fcf4c016600 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:16:38.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:38.747+0000 7fcf5effd700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c00f560 msgr2=0x7fcf4c016d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:16:38.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:38.747+0000 7fcf5effd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c00f560 0x7fcf4c016d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:16:38.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:38.747+0000 7fcf5effd700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008be0 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:16:38.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:38.747+0000 7fcf5effd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008be0 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:16:38.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:38.747+0000 7fcf5effd700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c0122b0 legacy=0x7fcf4c016600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:16:38.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:38.747+0000 7fcf5effd700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c015ae0 0x7fcf4c012bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:16:38.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:38.747+0000 7fcf5effd700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf680be640 con 0x7fcf4c00f970 2023-10-18T16:16:38.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:38.747+0000 7fcf5effd700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf6814d2b0 con 0x7fcf4c015ae0 2023-10-18T16:16:38.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:38.747+0000 7fcf6d0b0700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c00f970 legacy=0x7fcf4c005020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:16:38.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:38.747+0000 7fcf6c8af700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c015ae0 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:16:38.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:38.747+0000 7fcf6c8af700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c015ae0 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:16:38.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:38.948+0000 7fcf6d0b0700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c00f970 legacy=0x7fcf4c005020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:16:38.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:38.948+0000 7fcf6c8af700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c015ae0 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:16:38.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:38.948+0000 7fcf6c8af700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c015ae0 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:16:39.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:39.349+0000 7fcf6d0b0700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c00f970 legacy=0x7fcf4c005020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:16:39.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:39.349+0000 7fcf6c8af700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c015ae0 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:16:39.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:39.349+0000 7fcf6c8af700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c015ae0 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:16:39.455 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-18T16:16:39.458 DEBUG:teuthology.orchestra.run.smithi151:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-18T16:16:40.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:40.150+0000 7fcf6d0b0700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c00f970 legacy=0x7fcf4c005020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:16:40.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:40.150+0000 7fcf6c8af700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c015ae0 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:16:40.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:40.150+0000 7fcf6c8af700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c015ae0 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:16:41.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:41.747+0000 7fcf5effd700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c015ae0 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:16:41.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:41.747+0000 7fcf5effd700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c015ae0 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:16:41.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:41.747+0000 7fcf5effd700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c00f970 legacy=0x7fcf4c005020 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:16:41.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:41.747+0000 7fcf6d8b1700 1 -- reap_dead start 2023-10-18T16:16:41.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:41.747+0000 7fcf5effd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008be0 0x7fcf4c009d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:16:41.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:41.747+0000 7fcf5effd700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c015ae0 0x7fcf4c012bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:16:41.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:41.747+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008be0 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:16:41.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:41.747+0000 7fcf5effd700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf680be360 con 0x7fcf4c00f560 2023-10-18T16:16:41.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:41.747+0000 7fcf5effd700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf6814d2b0 con 0x7fcf4c008be0 2023-10-18T16:16:41.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:41.747+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008be0 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:16:41.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:41.747+0000 7fcf5effd700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf4c010160 con 0x7fcf4c015ae0 2023-10-18T16:16:41.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:41.747+0000 7fcf6d8b1700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c00f560 legacy=0x7fcf4c005020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:16:41.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:41.747+0000 7fcf6c8af700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c015ae0 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:16:41.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:41.747+0000 7fcf6c8af700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c015ae0 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:16:41.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:41.948+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008be0 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:16:41.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:41.948+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008be0 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:16:41.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:41.948+0000 7fcf6c8af700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c015ae0 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:16:41.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:41.948+0000 7fcf6d8b1700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c00f560 legacy=0x7fcf4c005020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:16:41.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:41.948+0000 7fcf6c8af700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c015ae0 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:16:42.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:42.349+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008be0 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:16:42.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:42.349+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008be0 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:16:42.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:42.349+0000 7fcf6d8b1700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c00f560 legacy=0x7fcf4c005020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:16:42.353 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:42.349+0000 7fcf6c8af700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c015ae0 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:16:42.353 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:42.349+0000 7fcf6c8af700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c015ae0 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:16:43.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:43.150+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008be0 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:16:43.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:43.150+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008be0 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:16:43.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:43.150+0000 7fcf6c8af700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c015ae0 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:16:43.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:43.150+0000 7fcf6d8b1700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c00f560 legacy=0x7fcf4c005020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:16:43.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:43.150+0000 7fcf6c8af700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c015ae0 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:16:44.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:44.747+0000 7fcf5effd700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c015ae0 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:16:44.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:44.747+0000 7fcf5effd700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c015ae0 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:16:44.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:44.747+0000 7fcf5effd700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008be0 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:16:44.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:44.747+0000 7fcf5effd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008be0 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:16:44.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:44.747+0000 7fcf5effd700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c00f560 legacy=0x7fcf4c005020 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:16:44.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:44.747+0000 7fcf5effd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c0122b0 0x7fcf4c016600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:16:44.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:44.747+0000 7fcf5effd700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf680be640 con 0x7fcf4c00f970 2023-10-18T16:16:44.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:44.747+0000 7fcf5effd700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf6814d2b0 con 0x7fcf4c0122b0 2023-10-18T16:16:44.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:44.747+0000 7fcf6d0b0700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c00f970 legacy=0x7fcf4c015ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:16:44.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:44.747+0000 7fcf6c8af700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c0122b0 msgr2=0x7fcf4c016600 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:16:44.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:44.747+0000 7fcf6c8af700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c0122b0 0x7fcf4c016600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:16:44.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:44.948+0000 7fcf6d0b0700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c00f970 legacy=0x7fcf4c015ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:16:44.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:44.948+0000 7fcf6c8af700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c0122b0 msgr2=0x7fcf4c016600 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:16:44.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:44.948+0000 7fcf6c8af700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c0122b0 0x7fcf4c016600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:16:45.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:45.349+0000 7fcf6d0b0700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c00f970 legacy=0x7fcf4c015ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:16:45.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:45.349+0000 7fcf6c8af700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c0122b0 msgr2=0x7fcf4c016600 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:16:45.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:45.349+0000 7fcf6c8af700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c0122b0 0x7fcf4c016600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:16:46.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:46.150+0000 7fcf6d0b0700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c00f970 legacy=0x7fcf4c015ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:16:46.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:46.150+0000 7fcf6c8af700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c0122b0 msgr2=0x7fcf4c016600 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:16:46.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:46.150+0000 7fcf6c8af700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c0122b0 0x7fcf4c016600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:16:47.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:47.747+0000 7fcf5effd700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c0122b0 msgr2=0x7fcf4c016600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:16:47.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:47.747+0000 7fcf5effd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c0122b0 0x7fcf4c016600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:16:47.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:47.747+0000 7fcf5effd700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c00f970 legacy=0x7fcf4c015ef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:16:47.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:47.747+0000 7fcf6d8b1700 1 -- reap_dead start 2023-10-18T16:16:47.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:47.747+0000 7fcf5effd700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c015ae0 0x7fcf4c012bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:16:47.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:47.747+0000 7fcf5effd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c0122b0 0x7fcf4c016600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:16:47.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:47.747+0000 7fcf5effd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008be0 0x7fcf4c009d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-18T16:16:47.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:47.747+0000 7fcf5effd700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf6814d2b0 con 0x7fcf4c008be0 2023-10-18T16:16:47.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:47.747+0000 7fcf5effd700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf4c010160 con 0x7fcf4c015ae0 2023-10-18T16:16:47.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:47.747+0000 7fcf5effd700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf4c00a760 con 0x7fcf4c0122b0 2023-10-18T16:16:47.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:47.747+0000 7fcf6d8b1700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008be0 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:16:47.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:47.748+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c015ae0 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:16:47.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:47.748+0000 7fcf6c8af700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c0122b0 msgr2=0x7fcf4c016600 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:16:47.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:47.748+0000 7fcf6d8b1700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008be0 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:16:47.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:47.748+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c015ae0 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:16:47.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:47.748+0000 7fcf6c8af700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c0122b0 0x7fcf4c016600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-18T16:16:47.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:47.949+0000 7fcf6d8b1700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008be0 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:16:47.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:47.949+0000 7fcf6d8b1700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008be0 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:16:47.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:47.949+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c015ae0 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:16:47.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:47.949+0000 7fcf6c8af700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c0122b0 msgr2=0x7fcf4c016600 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:16:47.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:47.949+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c015ae0 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:16:47.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:47.949+0000 7fcf6c8af700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c0122b0 0x7fcf4c016600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-18T16:16:48.353 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:48.350+0000 7fcf6d8b1700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008be0 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:16:48.353 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:48.350+0000 7fcf6d8b1700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008be0 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:16:48.353 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:48.350+0000 7fcf6c8af700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c0122b0 msgr2=0x7fcf4c016600 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:16:48.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:48.350+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c015ae0 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:16:48.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:48.350+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c015ae0 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:16:48.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:48.350+0000 7fcf6c8af700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c0122b0 0x7fcf4c016600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-18T16:16:49.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:49.151+0000 7fcf6d8b1700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008be0 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2023-10-18T16:16:49.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:49.151+0000 7fcf6d8b1700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008be0 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:16:49.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:49.151+0000 7fcf6d0b0700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c015ae0 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-10-18T16:16:49.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:49.151+0000 7fcf6c8af700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c0122b0 msgr2=0x7fcf4c016600 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-10-18T16:16:49.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:49.151+0000 7fcf6d0b0700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c015ae0 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:16:49.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:49.151+0000 7fcf6c8af700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c0122b0 0x7fcf4c016600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-18T16:16:50.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:50.749+0000 7fcf5effd700 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c0122b0 msgr2=0x7fcf4c016600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:16:50.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:50.749+0000 7fcf5effd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf4c0122b0 0x7fcf4c016600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:16:50.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:50.749+0000 7fcf5effd700 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c015ae0 msgr2=0x7fcf4c012bd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:16:50.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:50.749+0000 7fcf5effd700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf4c015ae0 0x7fcf4c012bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:16:50.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:50.749+0000 7fcf5effd700 1 -- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008be0 msgr2=0x7fcf4c009d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-18T16:16:50.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:50.749+0000 7fcf5effd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf4c008be0 0x7fcf4c009d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-10-18T16:16:50.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:50.749+0000 7fcf5effd700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf680be360 con 0x7fcf4c00f560 2023-10-18T16:16:50.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:50.749+0000 7fcf5effd700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf680be640 con 0x7fcf4c00f970 2023-10-18T16:16:50.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:50.749+0000 7fcf6d0b0700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c00f560 legacy=0x7fcf4c005020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:16:50.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:50.749+0000 7fcf6c8af700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c00f970 legacy=0x7fcf4c016af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:16:50.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:50.950+0000 7fcf6d0b0700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c00f560 legacy=0x7fcf4c005020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:16:50.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:50.950+0000 7fcf6c8af700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c00f970 legacy=0x7fcf4c016af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:16:51.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:51.351+0000 7fcf6d0b0700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c00f560 legacy=0x7fcf4c005020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:16:51.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:51.351+0000 7fcf6c8af700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c00f970 legacy=0x7fcf4c016af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:16:52.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:52.152+0000 7fcf6d0b0700 1 -- >> v1:172.21.15.88:6789/0 conn(0x7fcf4c00f560 legacy=0x7fcf4c005020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2023-10-18T16:16:52.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-18T16:16:52.152+0000 7fcf6c8af700 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fcf4c00f970 legacy=0x7fcf4c016af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-10-18T16:16:53.652 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-10-18T16:16:53.653 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph.py", line 1905, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph.py", line 1482, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 3166, in wait_until_healthy while self.get_mon_health()['status'] != 'HEALTH_OK': File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 3158, in get_mon_health out = self.raw_cluster_cmd('health', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 1600, in raw_cluster_cmd p = self.run_cluster_cmd(args=args, stdout=stdout, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 1593, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi088 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph.py", line 1911, in task ctx.managers[config['cluster']].stop_pg_num_changes() File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 2286, in stop_pg_num_changes osd_dump = self.get_osd_dump_json() File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 2520, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 1600, in raw_cluster_cmd p = self.run_cluster_cmd(args=args, stdout=stdout, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 1593, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi088 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' 2023-10-18T16:16:53.654 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-18T16:16:53.655 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-18T16:16:53.656 INFO:teuthology.orchestra.run:waiting for 300 2023-10-18T16:16:53.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T16:16:53.658 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph.py", line 1905, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph.py", line 1482, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 3166, in wait_until_healthy while self.get_mon_health()['status'] != 'HEALTH_OK': File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 3158, in get_mon_health out = self.raw_cluster_cmd('health', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 1600, in raw_cluster_cmd p = self.run_cluster_cmd(args=args, stdout=stdout, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 1593, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi088 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph.py", line 1451, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph.py", line 1911, in task ctx.managers[config['cluster']].stop_pg_num_changes() File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 2286, in stop_pg_num_changes osd_dump = self.get_osd_dump_json() File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 2520, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 1600, in raw_cluster_cmd p = self.run_cluster_cmd(args=args, stdout=stdout, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 1593, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi088 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi088 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a' 2023-10-18T16:16:53.660 INFO:tasks.ceph.mds.a:Stopped 2023-10-18T16:16:53.661 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-18T16:16:53.661 INFO:teuthology.orchestra.run:waiting for 300 2023-10-18T16:16:53.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T16:16:53.663 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph.py", line 1905, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph.py", line 1482, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 3166, in wait_until_healthy while self.get_mon_health()['status'] != 'HEALTH_OK': File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 3158, in get_mon_health out = self.raw_cluster_cmd('health', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 1600, in raw_cluster_cmd p = self.run_cluster_cmd(args=args, stdout=stdout, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 1593, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi088 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph.py", line 1451, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph.py", line 1911, in task ctx.managers[config['cluster']].stop_pg_num_changes() File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 2286, in stop_pg_num_changes osd_dump = self.get_osd_dump_json() File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 2520, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 1600, in raw_cluster_cmd p = self.run_cluster_cmd(args=args, stdout=stdout, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 1593, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi088 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi088 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c' 2023-10-18T16:16:53.664 INFO:tasks.ceph.mds.c:Stopped 2023-10-18T16:16:53.664 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2023-10-18T16:16:53.665 INFO:teuthology.orchestra.run:waiting for 300 2023-10-18T16:16:53.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T16:16:53.666 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph.py", line 1905, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph.py", line 1482, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 3166, in wait_until_healthy while self.get_mon_health()['status'] != 'HEALTH_OK': File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 3158, in get_mon_health out = self.raw_cluster_cmd('health', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 1600, in raw_cluster_cmd p = self.run_cluster_cmd(args=args, stdout=stdout, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 1593, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi088 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph.py", line 1451, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph.py", line 1911, in task ctx.managers[config['cluster']].stop_pg_num_changes() File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 2286, in stop_pg_num_changes osd_dump = self.get_osd_dump_json() File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 2520, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 1600, in raw_cluster_cmd p = self.run_cluster_cmd(args=args, stdout=stdout, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 1593, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi088 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi088 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e' 2023-10-18T16:16:53.666 INFO:tasks.ceph.mds.e:Stopped 2023-10-18T16:16:53.667 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-18T16:16:53.668 INFO:teuthology.orchestra.run:waiting for 300 2023-10-18T16:16:53.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T16:16:53.669 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph.py", line 1905, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph.py", line 1482, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 3166, in wait_until_healthy while self.get_mon_health()['status'] != 'HEALTH_OK': File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 3158, in get_mon_health out = self.raw_cluster_cmd('health', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 1600, in raw_cluster_cmd p = self.run_cluster_cmd(args=args, stdout=stdout, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 1593, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi088 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph.py", line 1451, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph.py", line 1911, in task ctx.managers[config['cluster']].stop_pg_num_changes() File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 2286, in stop_pg_num_changes osd_dump = self.get_osd_dump_json() File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 2520, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 1600, in raw_cluster_cmd p = self.run_cluster_cmd(args=args, stdout=stdout, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 1593, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi088 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi151 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b' 2023-10-18T16:16:53.669 INFO:tasks.ceph.mds.b:Stopped 2023-10-18T16:16:53.670 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-18T16:16:53.670 INFO:teuthology.orchestra.run:waiting for 300 2023-10-18T16:16:53.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T16:16:53.672 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph.py", line 1905, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph.py", line 1482, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 3166, in wait_until_healthy while self.get_mon_health()['status'] != 'HEALTH_OK': File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 3158, in get_mon_health out = self.raw_cluster_cmd('health', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 1600, in raw_cluster_cmd p = self.run_cluster_cmd(args=args, stdout=stdout, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 1593, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi088 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph.py", line 1451, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph.py", line 1911, in task ctx.managers[config['cluster']].stop_pg_num_changes() File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 2286, in stop_pg_num_changes osd_dump = self.get_osd_dump_json() File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 2520, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 1600, in raw_cluster_cmd p = self.run_cluster_cmd(args=args, stdout=stdout, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 1593, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi088 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi151 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d' 2023-10-18T16:16:53.672 INFO:tasks.ceph.mds.d:Stopped 2023-10-18T16:16:53.673 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2023-10-18T16:16:53.674 INFO:teuthology.orchestra.run:waiting for 300 2023-10-18T16:16:53.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T16:16:53.675 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph.py", line 1905, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph.py", line 1482, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 3166, in wait_until_healthy while self.get_mon_health()['status'] != 'HEALTH_OK': File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 3158, in get_mon_health out = self.raw_cluster_cmd('health', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 1600, in raw_cluster_cmd p = self.run_cluster_cmd(args=args, stdout=stdout, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 1593, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi088 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph.py", line 1451, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph.py", line 1911, in task ctx.managers[config['cluster']].stop_pg_num_changes() File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 2286, in stop_pg_num_changes osd_dump = self.get_osd_dump_json() File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 2520, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 1600, in raw_cluster_cmd p = self.run_cluster_cmd(args=args, stdout=stdout, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 1593, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi088 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi151 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f' 2023-10-18T16:16:53.675 INFO:tasks.ceph.mds.f:Stopped 2023-10-18T16:16:53.676 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-18T16:16:53.676 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-10-18T16:16:53.677 INFO:teuthology.orchestra.run:waiting for 300 2023-10-18T16:16:53.677 INFO:tasks.ceph.osd.0:Stopped 2023-10-18T16:16:53.678 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-10-18T16:16:53.679 INFO:teuthology.orchestra.run:waiting for 300 2023-10-18T16:16:53.680 INFO:tasks.ceph.osd.1:Stopped 2023-10-18T16:16:53.680 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-10-18T16:16:53.681 INFO:teuthology.orchestra.run:waiting for 300 2023-10-18T16:16:53.682 INFO:tasks.ceph.osd.2:Stopped 2023-10-18T16:16:53.683 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-10-18T16:16:53.684 INFO:teuthology.orchestra.run:waiting for 300 2023-10-18T16:16:53.685 INFO:tasks.ceph.osd.3:Stopped 2023-10-18T16:16:53.686 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-10-18T16:16:53.687 INFO:teuthology.orchestra.run:waiting for 300 2023-10-18T16:16:53.688 INFO:tasks.ceph.osd.4:Stopped 2023-10-18T16:16:53.689 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-10-18T16:16:53.689 INFO:teuthology.orchestra.run:waiting for 300 2023-10-18T16:16:53.690 INFO:tasks.ceph.osd.5:Stopped 2023-10-18T16:16:53.690 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-10-18T16:16:53.691 INFO:teuthology.orchestra.run:waiting for 300 2023-10-18T16:16:53.691 INFO:tasks.ceph.osd.6:Stopped 2023-10-18T16:16:53.692 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-10-18T16:16:53.693 INFO:teuthology.orchestra.run:waiting for 300 2023-10-18T16:16:53.694 INFO:tasks.ceph.osd.7:Stopped 2023-10-18T16:16:53.695 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-18T16:16:53.696 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-10-18T16:16:53.697 INFO:teuthology.orchestra.run:waiting for 300 2023-10-18T16:16:53.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T16:16:53.698 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph.py", line 1905, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph.py", line 1482, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 3166, in wait_until_healthy while self.get_mon_health()['status'] != 'HEALTH_OK': File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 3158, in get_mon_health out = self.raw_cluster_cmd('health', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 1600, in raw_cluster_cmd p = self.run_cluster_cmd(args=args, stdout=stdout, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 1593, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi088 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph.py", line 1451, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph.py", line 1911, in task ctx.managers[config['cluster']].stop_pg_num_changes() File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 2286, in stop_pg_num_changes osd_dump = self.get_osd_dump_json() File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 2520, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 1600, in raw_cluster_cmd p = self.run_cluster_cmd(args=args, stdout=stdout, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 1593, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi088 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi088 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y' 2023-10-18T16:16:53.699 INFO:tasks.ceph.mgr.y:Stopped 2023-10-18T16:16:53.699 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-10-18T16:16:53.700 INFO:teuthology.orchestra.run:waiting for 300 2023-10-18T16:16:53.700 INFO:tasks.ceph.mgr.x:Stopped 2023-10-18T16:16:53.701 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-18T16:16:53.701 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-18T16:16:53.702 INFO:teuthology.orchestra.run:waiting for 300 2023-10-18T16:16:53.702 INFO:tasks.ceph.mon.a:Stopped 2023-10-18T16:16:53.703 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-10-18T16:16:53.703 INFO:teuthology.orchestra.run:waiting for 300 2023-10-18T16:16:53.704 INFO:tasks.ceph.mon.b:Stopped 2023-10-18T16:16:53.704 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-10-18T16:16:53.705 INFO:teuthology.orchestra.run:waiting for 300 2023-10-18T16:16:53.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T16:16:53.706 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph.py", line 1905, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph.py", line 1482, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 3166, in wait_until_healthy while self.get_mon_health()['status'] != 'HEALTH_OK': File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 3158, in get_mon_health out = self.raw_cluster_cmd('health', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 1600, in raw_cluster_cmd p = self.run_cluster_cmd(args=args, stdout=stdout, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 1593, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi088 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph.py", line 1451, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph.py", line 1911, in task ctx.managers[config['cluster']].stop_pg_num_changes() File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 2286, in stop_pg_num_changes osd_dump = self.get_osd_dump_json() File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 2520, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 1600, in raw_cluster_cmd p = self.run_cluster_cmd(args=args, stdout=stdout, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 1593, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi088 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi151 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c' 2023-10-18T16:16:53.707 INFO:tasks.ceph.mon.c:Stopped 2023-10-18T16:16:53.707 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-18T16:16:53.708 DEBUG:teuthology.orchestra.run.smithi088:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'Replacing daemon mds' | head -n 1 2023-10-18T16:16:53.743 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi088.front.sepia.ceph.com 2023-10-18T16:16:53.744 DEBUG:teuthology.orchestra.run.smithi088:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-10-18T16:16:53.913 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi088.front.sepia.ceph.com 2023-10-18T16:16:53.914 DEBUG:teuthology.orchestra.run.smithi088:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-10-18T16:16:53.992 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi088.front.sepia.ceph.com 2023-10-18T16:16:53.993 DEBUG:teuthology.orchestra.run.smithi088:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-10-18T16:16:54.078 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi088.front.sepia.ceph.com 2023-10-18T16:16:54.079 DEBUG:teuthology.orchestra.run.smithi088:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-10-18T16:16:54.139 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi151.front.sepia.ceph.com 2023-10-18T16:16:54.140 DEBUG:teuthology.orchestra.run.smithi151:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-10-18T16:16:54.272 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi151.front.sepia.ceph.com 2023-10-18T16:16:54.272 DEBUG:teuthology.orchestra.run.smithi151:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-10-18T16:16:54.399 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi151.front.sepia.ceph.com 2023-10-18T16:16:54.400 DEBUG:teuthology.orchestra.run.smithi151:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-10-18T16:16:54.475 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi151.front.sepia.ceph.com 2023-10-18T16:16:54.475 DEBUG:teuthology.orchestra.run.smithi151:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-10-18T16:16:54.544 INFO:tasks.ceph:Archiving mon data... 2023-10-18T16:16:54.546 DEBUG:teuthology.misc:Transferring archived files from smithi088:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/vshankar-2023-10-16_12:48:28-fs-wip-vshankar-testing-pacific-20231016.091648-testing-default-smithi/7429067/data/mon.a.tgz 2023-10-18T16:16:54.547 DEBUG:teuthology.orchestra.run.smithi088:> mktemp 2023-10-18T16:16:54.562 INFO:teuthology.orchestra.run.smithi088.stdout:/tmp/tmp.6yW7RlaSI7 2023-10-18T16:16:54.563 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.6yW7RlaSI7 2023-10-18T16:16:54.790 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 0666 /tmp/tmp.6yW7RlaSI7 2023-10-18T16:16:54.872 DEBUG:teuthology.orchestra.remote:smithi088:/tmp/tmp.6yW7RlaSI7 is 826KB 2023-10-18T16:16:54.980 DEBUG:teuthology.orchestra.run.smithi088:> rm -fr /tmp/tmp.6yW7RlaSI7 2023-10-18T16:16:54.995 DEBUG:teuthology.misc:Transferring archived files from smithi151:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/vshankar-2023-10-16_12:48:28-fs-wip-vshankar-testing-pacific-20231016.091648-testing-default-smithi/7429067/data/mon.b.tgz 2023-10-18T16:16:54.995 DEBUG:teuthology.orchestra.run.smithi151:> mktemp 2023-10-18T16:16:55.013 INFO:teuthology.orchestra.run.smithi151.stdout:/tmp/tmp.O5z1U8PbSL 2023-10-18T16:16:55.014 DEBUG:teuthology.orchestra.run.smithi151:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.O5z1U8PbSL 2023-10-18T16:16:55.319 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod 0666 /tmp/tmp.O5z1U8PbSL 2023-10-18T16:16:55.400 DEBUG:teuthology.orchestra.remote:smithi151:/tmp/tmp.O5z1U8PbSL is 1000KB 2023-10-18T16:16:55.503 DEBUG:teuthology.orchestra.run.smithi151:> rm -fr /tmp/tmp.O5z1U8PbSL 2023-10-18T16:16:55.518 DEBUG:teuthology.misc:Transferring archived files from smithi151:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/vshankar-2023-10-16_12:48:28-fs-wip-vshankar-testing-pacific-20231016.091648-testing-default-smithi/7429067/data/mon.c.tgz 2023-10-18T16:16:55.518 DEBUG:teuthology.orchestra.run.smithi151:> mktemp 2023-10-18T16:16:55.571 INFO:teuthology.orchestra.run.smithi151.stdout:/tmp/tmp.PSPEK1PWX0 2023-10-18T16:16:55.572 DEBUG:teuthology.orchestra.run.smithi151:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.PSPEK1PWX0 2023-10-18T16:16:55.834 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod 0666 /tmp/tmp.PSPEK1PWX0 2023-10-18T16:16:55.915 DEBUG:teuthology.orchestra.remote:smithi151:/tmp/tmp.PSPEK1PWX0 is 999KB 2023-10-18T16:16:56.022 DEBUG:teuthology.orchestra.run.smithi151:> rm -fr /tmp/tmp.PSPEK1PWX0 2023-10-18T16:16:56.036 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-18T16:16:56.037 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-18T16:16:56.039 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-18T16:16:56.099 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-10-18T16:16:56.100 DEBUG:teuthology.orchestra.run.smithi088:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-18T16:16:56.105 DEBUG:teuthology.orchestra.run.smithi151:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-18T16:16:56.128 INFO:teuthology.orchestra.run.smithi088.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-18T16:16:56.164 INFO:teuthology.orchestra.run.smithi151.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-18T16:16:56.169 INFO:tasks.ceph:Archiving crash dumps... 2023-10-18T16:16:56.172 DEBUG:teuthology.misc:Transferring archived files from smithi088:/var/lib/ceph/crash to /home/teuthworker/archive/vshankar-2023-10-16_12:48:28-fs-wip-vshankar-testing-pacific-20231016.091648-testing-default-smithi/7429067/remote/smithi088/crash 2023-10-18T16:16:56.173 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-18T16:16:56.212 DEBUG:teuthology.misc:Transferring archived files from smithi151:/var/lib/ceph/crash to /home/teuthworker/archive/vshankar-2023-10-16_12:48:28-fs-wip-vshankar-testing-pacific-20231016.091648-testing-default-smithi/7429067/remote/smithi151/crash 2023-10-18T16:16:56.213 DEBUG:teuthology.orchestra.run.smithi151:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-18T16:16:56.252 INFO:tasks.ceph:Shutting down logrotate 2023-10-18T16:16:56.253 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-18T16:16:56.255 DEBUG:teuthology.orchestra.run.smithi151:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-18T16:16:56.335 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-10-18T16:16:56.359 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-10-18T16:16:56.382 INFO:tasks.ceph:Compressing logs... 2023-10-18T16:16:56.382 DEBUG:teuthology.orchestra.run.smithi088:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-18T16:16:56.402 DEBUG:teuthology.orchestra.run.smithi151:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-18T16:16:56.457 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91289.log 2023-10-18T16:16:56.457 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91200.log 2023-10-18T16:16:56.457 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91158.log 2023-10-18T16:16:56.457 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.91289.log: /var/log/ceph/ceph-client.admin.91200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91116.log 2023-10-18T16:16:56.458 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.91158.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.91289.log.gz 2023-10-18T16:16:56.458 INFO:teuthology.orchestra.run.smithi151.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.91200.log.gz 2023-10-18T16:16:56.458 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-10-18T16:16:56.458 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.91116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91416.log 2023-10-18T16:16:56.458 INFO:teuthology.orchestra.run.smithi151.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.91158.log.gz 2023-10-18T16:16:56.459 INFO:teuthology.orchestra.run.smithi151.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.91116.log.gz 2023-10-18T16:16:56.459 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2023-10-18T16:16:56.459 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.91416.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.f.log 2023-10-18T16:16:56.459 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-10-18T16:16:56.459 INFO:teuthology.orchestra.run.smithi151.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.91416.log.gz 2023-10-18T16:16:56.460 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-10-18T16:16:56.460 INFO:teuthology.orchestra.run.smithi151.stderr: 93.0% -- replaced with /var/log/ceph/ceph.log.gz 2023-10-18T16:16:56.460 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2023-10-18T16:16:56.460 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2023-10-18T16:16:56.460 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-mgr.x.log: /var/log/ceph/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91374.log 2023-10-18T16:16:56.461 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2023-10-18T16:16:56.461 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.91374.log: 93.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-18T16:16:56.461 INFO:teuthology.orchestra.run.smithi151.stderr: 87.8% -- replaced with /var/log/ceph/ceph-mds.f.log.gz 2023-10-18T16:16:56.461 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-mon.c.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.91374.log.gz 2023-10-18T16:16:56.461 INFO:teuthology.orchestra.run.smithi151.stderr: 86.5% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2023-10-18T16:16:56.461 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2023-10-18T16:16:56.462 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91240.log 2023-10-18T16:16:56.462 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2023-10-18T16:16:56.462 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.91240.log: /var/log/ceph/ceph-osd.6.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.91240.log.gz 2023-10-18T16:16:56.462 INFO:teuthology.orchestra.run.smithi151.stderr: 88.4% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2023-10-18T16:16:56.463 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2023-10-18T16:16:56.463 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91331.log 2023-10-18T16:16:56.463 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2023-10-18T16:16:56.463 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.91331.log: /var/log/ceph/ceph-mon.b.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.91331.log.gz 2023-10-18T16:16:56.475 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99271.log 2023-10-18T16:16:56.475 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97346.log 2023-10-18T16:16:56.475 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100237.log 2023-10-18T16:16:56.475 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.99271.log: /var/log/ceph/ceph-client.admin.97346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100153.log 2023-10-18T16:16:56.476 INFO:teuthology.orchestra.run.smithi088.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.99271.log.gz 2023-10-18T16:16:56.476 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.100237.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.97346.log.gz 2023-10-18T16:16:56.476 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100464.log 2023-10-18T16:16:56.476 INFO:teuthology.orchestra.run.smithi088.stderr: 88.0%/var/log/ceph/ceph-client.admin.100153.log: -- replaced with /var/log/ceph/ceph-client.admin.100237.log.gz 2023-10-18T16:16:56.476 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99708.log 2023-10-18T16:16:56.477 INFO:teuthology.orchestra.run.smithi088.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.100153.log.gz 2023-10-18T16:16:56.477 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.100464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95287.log 2023-10-18T16:16:56.477 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.99708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97750.log 2023-10-18T16:16:56.477 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.95287.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.99708.log.gz 2023-10-18T16:16:56.477 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95875.log 2023-10-18T16:16:56.478 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.97750.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.95287.log.gz 2023-10-18T16:16:56.478 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94424.log 2023-10-18T16:16:56.478 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.95875.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.97750.log.gz 2023-10-18T16:16:56.478 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99842.log 2023-10-18T16:16:56.478 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.94424.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.95875.log.gz 2023-10-18T16:16:56.479 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98386.log 2023-10-18T16:16:56.479 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.99842.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.94424.log.gz 2023-10-18T16:16:56.479 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91138.log 2023-10-18T16:16:56.479 INFO:teuthology.orchestra.run.smithi088.stderr: 95.4% -- replaced with /var/log/ceph/ceph-client.admin.100464.log.gz 2023-10-18T16:16:56.479 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.98386.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.99842.log.gz 2023-10-18T16:16:56.479 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100380.log 2023-10-18T16:16:56.480 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.91138.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.98386.log.gz 2023-10-18T16:16:56.480 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97135.log 2023-10-18T16:16:56.480 INFO:teuthology.orchestra.run.smithi088.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.91138.log.gz 2023-10-18T16:16:56.480 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.100380.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2023-10-18T16:16:56.480 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.97135.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.100380.log.gz 2023-10-18T16:16:56.481 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96782.log 2023-10-18T16:16:56.481 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-mds.a.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.97135.log.gz 2023-10-18T16:16:56.481 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96505.log 2023-10-18T16:16:56.481 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.96782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99355.log 2023-10-18T16:16:56.482 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.96505.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.96782.log.gz 2023-10-18T16:16:56.482 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99145.log 2023-10-18T16:16:56.482 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.99355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97581.log 88.2% 2023-10-18T16:16:56.482 INFO:teuthology.orchestra.run.smithi088.stderr: -- replaced with /var/log/ceph/ceph-client.admin.96505.log.gz 2023-10-18T16:16:56.482 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.99145.log: gzip 88.7%-5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.99355.log.gz-- 2023-10-18T16:16:56.483 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.96127.log 2023-10-18T16:16:56.483 INFO:teuthology.orchestra.run.smithi088.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.99145.log.gz 2023-10-18T16:16:56.483 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.97581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95371.log 2023-10-18T16:16:56.483 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.96127.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.97581.log.gz 2023-10-18T16:16:56.483 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95749.log 2023-10-18T16:16:56.484 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.95371.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.96127.log.gz 2023-10-18T16:16:56.484 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98428.log 2023-10-18T16:16:56.484 INFO:teuthology.orchestra.run.smithi088.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.95371.log.gz 2023-10-18T16:16:56.484 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.95749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97835.log 2023-10-18T16:16:56.484 INFO:teuthology.orchestra.run.smithi088.stderr: 89.1% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2023-10-18T16:16:56.485 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.98428.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.95749.log.gz 2023-10-18T16:16:56.485 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96824.log 2023-10-18T16:16:56.485 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.97835.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.98428.log.gz 2023-10-18T16:16:56.485 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99439.log 2023-10-18T16:16:56.485 INFO:teuthology.orchestra.run.smithi088.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.97835.log.gz 2023-10-18T16:16:56.486 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95917.log 2023-10-18T16:16:56.486 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.96824.log: /var/log/ceph/ceph-client.admin.99439.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98739.log 2023-10-18T16:16:56.486 INFO:teuthology.orchestra.run.smithi088.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.96824.log.gz 2023-10-18T16:16:56.486 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.95917.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.99439.log.gz 2023-10-18T16:16:56.486 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98823.log 2023-10-18T16:16:56.487 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.98739.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.95917.log.gz 2023-10-18T16:16:56.487 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98344.log 2023-10-18T16:16:56.487 INFO:teuthology.orchestra.run.smithi088.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.98739.log.gz 2023-10-18T16:16:56.487 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96295.log /var/log/ceph/ceph-client.admin.98823.log: 2023-10-18T16:16:56.487 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.98344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100338.log 2023-10-18T16:16:56.488 INFO:teuthology.orchestra.run.smithi088.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.98823.log.gz 2023-10-18T16:16:56.488 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.96295.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-10-18T16:16:56.488 INFO:teuthology.orchestra.run.smithi088.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.98344.log.gz 2023-10-18T16:16:56.488 INFO:teuthology.orchestra.run.smithi088.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.96295.log.gz 2023-10-18T16:16:56.488 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.100338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99624.log 2023-10-18T16:16:56.488 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96211.log 2023-10-18T16:16:56.489 INFO:teuthology.orchestra.run.smithi088.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.100338.log.gz 2023-10-18T16:16:56.489 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.99624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99187.log 2023-10-18T16:16:56.489 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.96211.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.99624.log.gz 2023-10-18T16:16:56.489 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97304.log 2023-10-18T16:16:56.489 INFO:teuthology.orchestra.run.smithi088.stderr: 87.4% -- replaced with /var/log/ceph/ceph.log.gz 2023-10-18T16:16:56.490 INFO:teuthology.orchestra.run.smithi088.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.96211.log.gz 2023-10-18T16:16:56.490 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.99187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99044.log 2023-10-18T16:16:56.490 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.97304.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.99187.log.gz 2023-10-18T16:16:56.490 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97978.log 2023-10-18T16:16:56.490 INFO:teuthology.orchestra.run.smithi088.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.97304.log.gz 2023-10-18T16:16:56.491 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.99044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97623.log 2023-10-18T16:16:56.491 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.97978.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.99044.log.gz 2023-10-18T16:16:56.491 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95245.log 2023-10-18T16:16:56.491 INFO:teuthology.orchestra.run.smithi088.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.97978.log.gz 2023-10-18T16:16:56.491 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.97623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96379.log 2023-10-18T16:16:56.492 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.95245.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.97623.log.gz 2023-10-18T16:16:56.492 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95707.log 2023-10-18T16:16:56.492 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.96379.log: gzip -5 --verbose -- 89.1%/var/log/ceph/ceph-client.admin.99666.log 2023-10-18T16:16:56.492 INFO:teuthology.orchestra.run.smithi088.stderr: -- replaced with /var/log/ceph/ceph-client.admin.95245.log.gz 2023-10-18T16:16:56.493 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.95707.log: 87.4%gzip -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.96379.log.gz-- 2023-10-18T16:16:56.493 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.97430.log 2023-10-18T16:16:56.493 INFO:teuthology.orchestra.run.smithi088.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.95707.log.gz 2023-10-18T16:16:56.493 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.99666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94521.log 2023-10-18T16:16:56.493 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.97430.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.99666.log.gz 2023-10-18T16:16:56.494 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97034.log 2023-10-18T16:16:56.494 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.94521.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.84581.log 2023-10-18T16:16:56.494 INFO:teuthology.orchestra.run.smithi088.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.97430.log.gz 2023-10-18T16:16:56.494 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.97034.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.94521.log.gz 2023-10-18T16:16:56.494 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99313.log 2023-10-18T16:16:56.494 INFO:teuthology.orchestra.run.smithi088.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.97034.log.gz 2023-10-18T16:16:56.495 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph.tmp-client.admin.84581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96908.log 2023-10-18T16:16:56.495 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.84581.log.gz 2023-10-18T16:16:56.495 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.99313.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97177.log 2023-10-18T16:16:56.495 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.96908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98188.log 2023-10-18T16:16:56.496 INFO:teuthology.orchestra.run.smithi088.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.99313.log.gz 2023-10-18T16:16:56.496 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.97177.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.96908.log.gzgzip 2023-10-18T16:16:56.496 INFO:teuthology.orchestra.run.smithi088.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.99397.log 2023-10-18T16:16:56.496 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.98188.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.97177.log.gz 2023-10-18T16:16:56.496 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95581.log 2023-10-18T16:16:56.496 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.99397.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.98188.log.gz 2023-10-18T16:16:56.497 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94479.log 2023-10-18T16:16:56.497 INFO:teuthology.orchestra.run.smithi088.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.99397.log.gz 2023-10-18T16:16:56.497 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.95581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98908.log 2023-10-18T16:16:56.497 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.94479.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-10-18T16:16:56.497 INFO:teuthology.orchestra.run.smithi088.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.95581.log.gz 2023-10-18T16:16:56.498 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.98908.log: 87.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.94479.log.gz-5 2023-10-18T16:16:56.498 INFO:teuthology.orchestra.run.smithi088.stderr:--verbose -- /var/log/ceph/ceph-client.admin.99103.log 2023-10-18T16:16:56.498 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph.audit.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.98908.log.gz 2023-10-18T16:16:56.498 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96463.log 2023-10-18T16:16:56.498 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.99103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95413.log 2023-10-18T16:16:56.499 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.96463.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.99103.log.gz 2023-10-18T16:16:56.499 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96043.log 2023-10-18T16:16:56.499 INFO:teuthology.orchestra.run.smithi088.stderr: 89.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-18T16:16:56.499 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.95413.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.96463.log.gz 2023-10-18T16:16:56.499 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95161.log 2023-10-18T16:16:56.500 INFO:teuthology.orchestra.run.smithi088.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.95413.log.gz 2023-10-18T16:16:56.500 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.96043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96085.log 2023-10-18T16:16:56.500 INFO:teuthology.orchestra.run.smithi088.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.96043.log.gz 2023-10-18T16:16:56.500 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.95161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98638.log 2023-10-18T16:16:56.500 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.96085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96740.log 2023-10-18T16:16:56.501 INFO:teuthology.orchestra.run.smithi088.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.95161.log.gz 2023-10-18T16:16:56.501 INFO:teuthology.orchestra.run.smithi088.stderr: 88.4%/var/log/ceph/ceph-client.admin.98638.log: -- replaced with /var/log/ceph/ceph-client.admin.96085.log.gz 2023-10-18T16:16:56.501 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98104.log 2023-10-18T16:16:56.501 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.96740.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.98638.log.gz 2023-10-18T16:16:56.501 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94717.log 2023-10-18T16:16:56.502 INFO:teuthology.orchestra.run.smithi088.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.96740.log.gz 2023-10-18T16:16:56.502 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.98104.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94946.log 2023-10-18T16:16:56.502 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.94717.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 88.0% 2023-10-18T16:16:56.502 INFO:teuthology.orchestra.run.smithi088.stderr: -- replaced with /var/log/ceph/ceph-client.admin.98104.log.gz 2023-10-18T16:16:56.502 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.94946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94988.log 2023-10-18T16:16:56.503 INFO:teuthology.orchestra.run.smithi088.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.94717.log.gz 2023-10-18T16:16:56.503 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98235.log 2023-10-18T16:16:56.503 INFO:teuthology.orchestra.run.smithi088.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.94946.log.gz 2023-10-18T16:16:56.503 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.94988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95075.log 2023-10-18T16:16:56.503 INFO:teuthology.orchestra.run.smithi088.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.94988.log.gz 2023-10-18T16:16:56.503 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.98235.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97219.log 2023-10-18T16:16:56.504 INFO:teuthology.orchestra.run.smithi088.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.98235.log.gz 2023-10-18T16:16:56.504 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.95075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97792.log 2023-10-18T16:16:56.504 INFO:teuthology.orchestra.run.smithi088.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.95075.log.gz 2023-10-18T16:16:56.504 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.97219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96001.log 2023-10-18T16:16:56.504 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.97792.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.97219.log.gz 2023-10-18T16:16:56.505 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96631.log 2023-10-18T16:16:56.505 INFO:teuthology.orchestra.run.smithi088.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.97792.log.gz 2023-10-18T16:16:56.505 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.96001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99498.log 2023-10-18T16:16:56.505 INFO:teuthology.orchestra.run.smithi088.stderr: 88.8% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2023-10-18T16:16:56.505 INFO:teuthology.orchestra.run.smithi088.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.96001.log.gz 2023-10-18T16:16:56.506 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.96631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95665.log 2023-10-18T16:16:56.506 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.99498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95329.log 2023-10-18T16:16:56.506 INFO:teuthology.orchestra.run.smithi088.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.96631.log.gz 2023-10-18T16:16:56.506 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.95665.log: 88.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.99498.log.gz-5 2023-10-18T16:16:56.506 INFO:teuthology.orchestra.run.smithi088.stderr:--verbose -- /var/log/ceph/ceph-client.admin.98554.log 2023-10-18T16:16:56.507 INFO:teuthology.orchestra.run.smithi088.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.95665.log.gz 2023-10-18T16:16:56.507 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.95329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99901.log 2023-10-18T16:16:56.507 INFO:teuthology.orchestra.run.smithi088.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.95329.log.gz 2023-10-18T16:16:56.507 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.98554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97076.log 2023-10-18T16:16:56.507 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.99901.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.98554.log.gz 2023-10-18T16:16:56.508 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100111.log 2023-10-18T16:16:56.508 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.97076.log: 88.8%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.99901.log.gz--verbose 2023-10-18T16:16:56.508 INFO:teuthology.orchestra.run.smithi088.stderr:-- /var/log/ceph/ceph-client.admin.97262.log 2023-10-18T16:16:56.508 INFO:teuthology.orchestra.run.smithi088.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.97076.log.gz 2023-10-18T16:16:56.508 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.100111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98470.log 2023-10-18T16:16:56.509 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.97262.log: gzip 88.9%-5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.100111.log.gz-- 2023-10-18T16:16:56.509 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.96950.log 2023-10-18T16:16:56.509 INFO:teuthology.orchestra.run.smithi088.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.97262.log.gz 2023-10-18T16:16:56.509 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.98470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94845.log 2023-10-18T16:16:56.509 INFO:teuthology.orchestra.run.smithi088.stderr: 89.1%/var/log/ceph/ceph-client.admin.96950.log: -- replaced with /var/log/ceph/ceph-client.admin.98470.log.gz 2023-10-18T16:16:56.509 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99582.log 2023-10-18T16:16:56.510 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.94845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96421.log 2023-10-18T16:16:56.510 INFO:teuthology.orchestra.run.smithi088.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.96950.log.gz 2023-10-18T16:16:56.510 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.99582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96681.log 2023-10-18T16:16:56.510 INFO:teuthology.orchestra.run.smithi088.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.94845.log.gz 2023-10-18T16:16:56.510 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.96421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95497.log 2023-10-18T16:16:56.511 INFO:teuthology.orchestra.run.smithi088.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.99582.log.gz 2023-10-18T16:16:56.511 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.96681.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.96421.log.gz 2023-10-18T16:16:56.511 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2023-10-18T16:16:56.511 INFO:teuthology.orchestra.run.smithi088.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.96681.log.gz 2023-10-18T16:16:56.511 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.95497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100422.log 2023-10-18T16:16:56.512 INFO:teuthology.orchestra.run.smithi088.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.95497.log.gz 2023-10-18T16:16:56.512 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-10-18T16:16:56.512 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.100422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99229.log 2023-10-18T16:16:56.512 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98994.log 2023-10-18T16:16:56.512 INFO:teuthology.orchestra.run.smithi088.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.100422.log.gz 2023-10-18T16:16:56.513 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.99229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99943.log 2023-10-18T16:16:56.513 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.98994.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.99229.log.gz 2023-10-18T16:16:56.513 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98781.log 2023-10-18T16:16:56.513 INFO:teuthology.orchestra.run.smithi088.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.98994.log.gz 2023-10-18T16:16:56.513 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95030.log 2023-10-18T16:16:56.514 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.99943.log: /var/log/ceph/ceph-client.admin.98781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98952.log 2023-10-18T16:16:56.514 INFO:teuthology.orchestra.run.smithi088.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.99943.log.gz 2023-10-18T16:16:56.514 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.95030.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.98781.log.gz 2023-10-18T16:16:56.514 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100296.log 2023-10-18T16:16:56.514 INFO:teuthology.orchestra.run.smithi088.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.95030.log.gz 2023-10-18T16:16:56.515 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98865.log 2023-10-18T16:16:56.515 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.98952.log: /var/log/ceph/ceph-client.admin.100296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96337.log 2023-10-18T16:16:56.515 INFO:teuthology.orchestra.run.smithi088.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.98952.log.gz 2023-10-18T16:16:56.515 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.98865.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.100296.log.gz 2023-10-18T16:16:56.515 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-10-18T16:16:56.516 INFO:teuthology.orchestra.run.smithi088.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.98865.log.gz 2023-10-18T16:16:56.516 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.96337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95959.log 2023-10-18T16:16:56.516 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98512.log 2023-10-18T16:16:56.516 INFO:teuthology.orchestra.run.smithi088.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.96337.log.gz 2023-10-18T16:16:56.516 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.95959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97936.log 2023-10-18T16:16:56.517 INFO:teuthology.orchestra.run.smithi088.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.95959.log.gz 2023-10-18T16:16:56.517 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.98512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98596.log 2023-10-18T16:16:56.517 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100027.log 2023-10-18T16:16:56.517 INFO:teuthology.orchestra.run.smithi088.stderr: 88.3%/var/log/ceph/ceph-client.admin.97936.log: -- replaced with /var/log/ceph/ceph-client.admin.98512.log.gz 2023-10-18T16:16:56.517 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.98596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90609.log 2023-10-18T16:16:56.518 INFO:teuthology.orchestra.run.smithi088.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.97936.log.gz 2023-10-18T16:16:56.518 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.100027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95539.log 2023-10-18T16:16:56.518 INFO:teuthology.orchestra.run.smithi088.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.98596.log.gz 2023-10-18T16:16:56.518 INFO:teuthology.orchestra.run.smithi088.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.100027.log.gz 2023-10-18T16:16:56.518 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.90609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96547.log 2023-10-18T16:16:56.519 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.95539.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94904.log 2023-10-18T16:16:56.519 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.96547.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97388.log 2023-10-18T16:16:56.519 INFO:teuthology.orchestra.run.smithi088.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.95539.log.gz 2023-10-18T16:16:56.519 INFO:teuthology.orchestra.run.smithi088.stderr: 93.3% -- replaced with /var/log/ceph/ceph-client.admin.90609.log.gz 2023-10-18T16:16:56.519 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.94904.log: 88.7%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.96547.log.gz--verbose 2023-10-18T16:16:56.520 INFO:teuthology.orchestra.run.smithi088.stderr:-- /var/log/ceph/ceph-client.admin.95833.log 2023-10-18T16:16:56.520 INFO:teuthology.orchestra.run.smithi088.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.94904.log.gz 2023-10-18T16:16:56.520 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.97388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99750.log 2023-10-18T16:16:56.520 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94801.log 2023-10-18T16:16:56.520 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.95833.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.97388.log.gz 2023-10-18T16:16:56.520 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.99750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96169.log 2023-10-18T16:16:56.521 INFO:teuthology.orchestra.run.smithi088.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.95833.log.gz 2023-10-18T16:16:56.521 INFO:teuthology.orchestra.run.smithi088.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.99750.log.gz 2023-10-18T16:16:56.521 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.94801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99792.log 2023-10-18T16:16:56.521 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.96169.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.94801.log.gz 2023-10-18T16:16:56.521 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95623.log 2023-10-18T16:16:56.521 INFO:teuthology.orchestra.run.smithi088.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.96169.log.gz 2023-10-18T16:16:56.522 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.99792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99540.log 2023-10-18T16:16:56.522 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.95623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97708.log 2023-10-18T16:16:56.522 INFO:teuthology.orchestra.run.smithi088.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.99792.log.gz 2023-10-18T16:16:56.522 INFO:teuthology.orchestra.run.smithi088.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.95623.log.gz 2023-10-18T16:16:56.522 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.99540.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-10-18T16:16:56.522 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.97708.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.99540.log.gz 2023-10-18T16:16:56.523 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98285.log 2023-10-18T16:16:56.523 INFO:teuthology.orchestra.run.smithi088.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.97708.log.gz 2023-10-18T16:16:56.523 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91006.log 2023-10-18T16:16:56.523 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97539.log 2023-10-18T16:16:56.523 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.98285.log: /var/log/ceph/ceph-client.admin.91006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100069.log 2023-10-18T16:16:56.524 INFO:teuthology.orchestra.run.smithi088.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.98285.log.gz 2023-10-18T16:16:56.524 INFO:teuthology.orchestra.run.smithi088.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.91006.log.gz 2023-10-18T16:16:56.524 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.97539.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95455.log 2023-10-18T16:16:56.524 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.100069.log: gzip 88.7%-5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.97539.log.gz-- 2023-10-18T16:16:56.524 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.95203.log 2023-10-18T16:16:56.524 INFO:teuthology.orchestra.run.smithi088.stderr: 88.0%/var/log/ceph/ceph-client.admin.95455.log: -- replaced with /var/log/ceph/ceph-client.admin.100069.log.gz 2023-10-18T16:16:56.525 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97480.log 2023-10-18T16:16:56.525 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.95203.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.95455.log.gz 2023-10-18T16:16:56.525 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96253.log 2023-10-18T16:16:56.525 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.97480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96866.log 2023-10-18T16:16:56.526 INFO:teuthology.orchestra.run.smithi088.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.95203.log.gz 2023-10-18T16:16:56.526 INFO:teuthology.orchestra.run.smithi088.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.97480.log.gz 2023-10-18T16:16:56.526 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.96253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97877.log 2023-10-18T16:16:56.526 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.96866.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.96253.log.gz 2023-10-18T16:16:56.526 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100195.log 2023-10-18T16:16:56.527 INFO:teuthology.orchestra.run.smithi088.stderr:gzip 89.0%/var/log/ceph/ceph-client.admin.97877.log: -5 -- replaced with /var/log/ceph/ceph-client.admin.96866.log.gz--verbose 2023-10-18T16:16:56.527 INFO:teuthology.orchestra.run.smithi088.stderr:-- /var/log/ceph/ceph-client.admin.98146.log 2023-10-18T16:16:56.527 INFO:teuthology.orchestra.run.smithi088.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.97877.log.gz 2023-10-18T16:16:56.527 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.100195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94759.log 2023-10-18T16:16:56.527 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98020.log 2023-10-18T16:16:56.528 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.98146.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.100195.log.gz 2023-10-18T16:16:56.528 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.94759.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.98146.log.gz 2023-10-18T16:16:56.528 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.98020.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.94759.log.gz 2023-10-18T16:16:56.528 INFO:teuthology.orchestra.run.smithi088.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.98020.log.gz 2023-10-18T16:16:56.528 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2023-10-18T16:16:56.529 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96589.log 2023-10-18T16:16:56.529 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95791.log 2023-10-18T16:16:56.529 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.96589.log: /var/log/ceph/ceph-client.admin.95791.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.96589.log.gz 2023-10-18T16:16:56.529 INFO:teuthology.orchestra.run.smithi088.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.95791.log.gz 2023-10-18T16:16:56.529 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94617.log 2023-10-18T16:16:56.530 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98062.log 2023-10-18T16:16:56.530 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.94617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97666.log 2023-10-18T16:16:56.530 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.98062.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.94617.log.gz 2023-10-18T16:16:56.530 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.97666.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.98062.log.gz 2023-10-18T16:16:56.530 INFO:teuthology.orchestra.run.smithi088.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.97666.log.gz 2023-10-18T16:16:56.531 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96992.log 2023-10-18T16:16:56.531 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99985.log 2023-10-18T16:16:56.531 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.96992.log: /var/log/ceph/ceph-client.admin.99985.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.96992.log.gz 2023-10-18T16:16:56.531 INFO:teuthology.orchestra.run.smithi088.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.99985.log.gz 2023-10-18T16:16:56.531 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.e.log 2023-10-18T16:16:56.532 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95119.log 2023-10-18T16:16:56.532 INFO:teuthology.orchestra.run.smithi088.stderr: 93.8% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2023-10-18T16:16:56.532 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100579.log 2023-10-18T16:16:56.532 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.95119.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-10-18T16:16:56.532 INFO:teuthology.orchestra.run.smithi088.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.95119.log.gz/var/log/ceph/ceph-client.admin.100579.log: 2023-10-18T16:16:56.533 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98697.log 2023-10-18T16:16:56.533 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.98697.log: 88.4% -- replaced with /var/log/ceph/ceph-mds.e.log.gz 2023-10-18T16:16:56.533 INFO:teuthology.orchestra.run.smithi088.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.98697.log.gz 2023-10-18T16:16:56.533 INFO:teuthology.orchestra.run.smithi088.stderr: 95.5% -- replaced with /var/log/ceph/ceph-client.admin.100579.log.gz 2023-10-18T16:16:56.534 INFO:teuthology.orchestra.run.smithi151.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-10-18T16:16:56.560 INFO:teuthology.orchestra.run.smithi151.stderr: 92.1% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2023-10-18T16:16:56.599 INFO:teuthology.orchestra.run.smithi088.stderr: 92.7% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-10-18T16:16:56.605 INFO:teuthology.orchestra.run.smithi088.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-10-18T16:16:56.606 INFO:teuthology.orchestra.run.smithi151.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2023-10-18T16:16:56.614 INFO:teuthology.orchestra.run.smithi151.stderr: 91.9% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2023-10-18T16:16:56.616 INFO:teuthology.orchestra.run.smithi151.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2023-10-18T16:16:56.637 INFO:teuthology.orchestra.run.smithi151.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2023-10-18T16:16:56.645 INFO:teuthology.orchestra.run.smithi151.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2023-10-18T16:16:56.647 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-10-18T16:16:56.647 INFO:teuthology.orchestra.run.smithi151.stderr:real 0m0.211s 2023-10-18T16:16:56.647 INFO:teuthology.orchestra.run.smithi151.stderr:user 0m1.012s 2023-10-18T16:16:56.647 INFO:teuthology.orchestra.run.smithi151.stderr:sys 0m0.050s 2023-10-18T16:16:56.656 INFO:teuthology.orchestra.run.smithi088.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-10-18T16:16:56.677 INFO:teuthology.orchestra.run.smithi088.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-10-18T16:16:56.681 INFO:teuthology.orchestra.run.smithi088.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2023-10-18T16:16:56.683 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-10-18T16:16:56.683 INFO:teuthology.orchestra.run.smithi088.stderr:real 0m0.270s 2023-10-18T16:16:56.684 INFO:teuthology.orchestra.run.smithi088.stderr:user 0m1.001s 2023-10-18T16:16:56.684 INFO:teuthology.orchestra.run.smithi088.stderr:sys 0m0.113s 2023-10-18T16:16:56.684 INFO:tasks.ceph:Archiving logs... 2023-10-18T16:16:56.685 DEBUG:teuthology.misc:Transferring archived files from smithi088:/var/log/ceph to /home/teuthworker/archive/vshankar-2023-10-16_12:48:28-fs-wip-vshankar-testing-pacific-20231016.091648-testing-default-smithi/7429067/remote/smithi088/log 2023-10-18T16:16:56.686 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-18T16:16:57.014 DEBUG:teuthology.misc:Transferring archived files from smithi151:/var/log/ceph to /home/teuthworker/archive/vshankar-2023-10-16_12:48:28-fs-wip-vshankar-testing-pacific-20231016.091648-testing-default-smithi/7429067/remote/smithi151/log 2023-10-18T16:16:57.016 DEBUG:teuthology.orchestra.run.smithi151:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-18T16:16:57.172 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph.py", line 1905, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph.py", line 1482, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 3166, in wait_until_healthy while self.get_mon_health()['status'] != 'HEALTH_OK': File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 3158, in get_mon_health out = self.raw_cluster_cmd('health', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 1600, in raw_cluster_cmd p = self.run_cluster_cmd(args=args, stdout=stdout, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 1593, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi088 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph.py", line 1911, in task ctx.managers[config['cluster']].stop_pg_num_changes() File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 2286, in stop_pg_num_changes osd_dump = self.get_osd_dump_json() File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 2520, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 1600, in raw_cluster_cmd p = self.run_cluster_cmd(args=args, stdout=stdout, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 1593, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi088 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' 2023-10-18T16:16:57.248 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=7a6392e832b64f928b10d2794f9bf3f7 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph.py", line 1905, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph.py", line 1482, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 3166, in wait_until_healthy while self.get_mon_health()['status'] != 'HEALTH_OK': File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 3158, in get_mon_health out = self.raw_cluster_cmd('health', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 1600, in raw_cluster_cmd p = self.run_cluster_cmd(args=args, stdout=stdout, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 1593, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi088 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph.py", line 1911, in task ctx.managers[config['cluster']].stop_pg_num_changes() File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 2286, in stop_pg_num_changes osd_dump = self.get_osd_dump_json() File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 2520, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 1600, in raw_cluster_cmd p = self.run_cluster_cmd(args=args, stdout=stdout, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks/ceph_manager.py", line 1593, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi088 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' 2023-10-18T16:16:57.255 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-18T16:16:57.278 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-18T16:16:57.305 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-10-18T16:16:57.306 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-18T16:16:57.320 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-18T16:16:57.391 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-10-18T16:16:57.401 INFO:teuthology.task.clock:Checking final clock skew... 2023-10-18T16:16:57.402 DEBUG:teuthology.orchestra.run.smithi088:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-18T16:16:57.406 DEBUG:teuthology.orchestra.run.smithi151:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-18T16:16:57.416 INFO:teuthology.orchestra.run.smithi088.stderr:bash: ntpq: command not found 2023-10-18T16:16:57.419 INFO:teuthology.orchestra.run.smithi151.stderr:bash: ntpq: command not found 2023-10-18T16:16:57.420 INFO:teuthology.orchestra.run.smithi088.stdout:210 Number of sources = 4 2023-10-18T16:16:57.420 INFO:teuthology.orchestra.run.smithi088.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-18T16:16:57.421 INFO:teuthology.orchestra.run.smithi088.stdout:=============================================================================== 2023-10-18T16:16:57.421 INFO:teuthology.orchestra.run.smithi088.stdout:^* hv01.front.sepia.ceph.com 3 6 377 25 +2035us[+2046us] +/- 75ms 2023-10-18T16:16:57.421 INFO:teuthology.orchestra.run.smithi088.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 27 -260us[ -249us] +/- 78ms 2023-10-18T16:16:57.422 INFO:teuthology.orchestra.run.smithi088.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 27 -1886us[-1875us] +/- 85ms 2023-10-18T16:16:57.422 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-10-18T16:16:57.424 INFO:teuthology.orchestra.run.smithi151.stdout:210 Number of sources = 4 2023-10-18T16:16:57.425 INFO:teuthology.orchestra.run.smithi151.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-18T16:16:57.425 INFO:teuthology.orchestra.run.smithi151.stdout:=============================================================================== 2023-10-18T16:16:57.426 INFO:teuthology.orchestra.run.smithi151.stdout:^* hv01.front.sepia.ceph.com 3 6 377 25 +1995us[+1998us] +/- 75ms 2023-10-18T16:16:57.426 INFO:teuthology.orchestra.run.smithi151.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 26 -306us[ -304us] +/- 78ms 2023-10-18T16:16:57.426 INFO:teuthology.orchestra.run.smithi151.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 26 -1948us[-1945us] +/- 85ms 2023-10-18T16:16:57.427 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-10-18T16:16:57.427 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-18T16:16:57.437 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-18T16:16:57.437 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-18T16:16:57.481 DEBUG:teuthology.orchestra.run.smithi088:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-18T16:16:57.572 DEBUG:teuthology.orchestra.run.smithi151:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-18T16:16:57.660 DEBUG:teuthology.orchestra.run.smithi088:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-18T16:16:57.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T16:16:57.690 DEBUG:teuthology.orchestra.run.smithi151:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-18T16:16:57.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T16:16:57.720 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-18T16:16:57.729 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-18T16:16:57.756 INFO:teuthology.task.internal:Duration was 1232.486735 seconds 2023-10-18T16:16:57.756 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-18T16:16:57.807 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-18T16:16:57.808 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-18T16:16:57.811 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-18T16:16:57.851 INFO:teuthology.orchestra.run.smithi088.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-18T16:16:57.853 INFO:teuthology.orchestra.run.smithi151.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-18T16:16:58.366 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-18T16:16:58.369 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi088.front.sepia.ceph.com 2023-10-18T16:16:58.370 DEBUG:teuthology.orchestra.run.smithi088:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-18T16:16:58.390 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi151.front.sepia.ceph.com 2023-10-18T16:16:58.390 DEBUG:teuthology.orchestra.run.smithi151:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-18T16:16:58.426 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-18T16:16:58.426 DEBUG:teuthology.orchestra.run.smithi088:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-18T16:16:58.432 DEBUG:teuthology.orchestra.run.smithi151:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-18T16:16:58.504 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-18T16:16:58.504 DEBUG:teuthology.orchestra.run.smithi088:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-18T16:16:58.512 DEBUG:teuthology.orchestra.run.smithi151:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-18T16:16:58.614 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-18T16:16:58.624 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-18T16:16:58.624 DEBUG:teuthology.orchestra.run.smithi088:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-18T16:16:58.649 DEBUG:teuthology.orchestra.run.smithi151:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-18T16:16:58.673 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-18T16:16:58.683 DEBUG:teuthology.orchestra.run.smithi088:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-18T16:16:58.691 DEBUG:teuthology.orchestra.run.smithi151:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-18T16:16:58.712 INFO:teuthology.orchestra.run.smithi088.stdout:kernel.core_pattern = core 2023-10-18T16:16:58.735 INFO:teuthology.orchestra.run.smithi151.stdout:kernel.core_pattern = core 2023-10-18T16:16:58.758 DEBUG:teuthology.orchestra.run.smithi088:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-18T16:16:58.788 WARNING:teuthology.task.internal:Found coredumps on ubuntu@smithi088.front.sepia.ceph.com, flagging run as failed 2023-10-18T16:16:58.788 DEBUG:teuthology.orchestra.run.smithi151:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-18T16:16:58.812 WARNING:teuthology.task.internal:Found coredumps on ubuntu@smithi151.front.sepia.ceph.com, flagging run as failed 2023-10-18T16:16:58.812 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-18T16:16:58.822 INFO:teuthology.task.internal:Transferring archived files... 2023-10-18T16:16:58.823 DEBUG:teuthology.misc:Transferring archived files from smithi088:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2023-10-16_12:48:28-fs-wip-vshankar-testing-pacific-20231016.091648-testing-default-smithi/7429067/remote/smithi088 2023-10-18T16:16:58.824 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-18T16:17:22.941 INFO:teuthology.task.internal:Transferring binaries for coredumps... 2023-10-18T16:17:22.952 INFO:teuthology.task.internal: core looks like: /home/teuthworker/archive/vshankar-2023-10-16_12:48:28-fs-wip-vshankar-testing-pacific-20231016.091648-testing-default-smithi/7429067/remote/smithi088/coredump/1697645373.94618.core.gz: gzip compressed data, was "1697645373.94618.core", last modified: Wed Oct 18 16:16:58 2023, max compression, original size modulo 2^32 300449792 2023-10-18T16:17:22.953 INFO:teuthology.task.internal:core is compressed, try accessing gzip file ... 2023-10-18T16:17:24.140 INFO:teuthology.task.internal: core looks like: /tmp/tmpxpj7mh36: ELF 64-bit LSB core file, x86-64, version 1 (SYSV), SVR4-style, from 'ceph-mds -f --cluster ceph -i a', real uid: 0, effective uid: 0, real gid: 0, effective gid: 0, execfn: '/bin/ceph-mds', platform: 'x86_64' 2023-10-18T16:17:24.174 INFO:teuthology.task.internal: dump_program: ceph-mds -f --cluster ceph -i a 2023-10-18T16:17:24.175 DEBUG:teuthology.orchestra.run.smithi088:> which 'ceph-mds -f --cluster ceph -i a' 2023-10-18T16:17:24.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T16:17:24.193 INFO:teuthology.orchestra.run.smithi088.stderr:which: no ceph-mds -f --cluster ceph -i a in (/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin) 2023-10-18T16:17:24.194 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/task/internal/__init__.py", line 420, in archive fetch_binaries_for_coredumps(path, rem) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/task/internal/__init__.py", line 345, in fetch_binaries_for_coredumps remote_path = remote.sh(['which', dump_program]).rstrip() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/remote.py", line 96, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi088 with status 1: "which 'ceph-mds -f --cluster ceph -i a'" 2023-10-18T16:17:24.203 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-18T16:17:24.215 INFO:teuthology.task.internal:Not uploading archives. 2023-10-18T16:17:24.216 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-18T16:17:24.263 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-18T16:17:24.264 DEBUG:teuthology.orchestra.run.smithi088:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-18T16:17:24.269 DEBUG:teuthology.orchestra.run.smithi151:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-18T16:17:24.282 INFO:teuthology.orchestra.run.smithi088.stdout: 262176 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Oct 18 16:16 /home/ubuntu/cephtest 2023-10-18T16:17:24.285 INFO:teuthology.orchestra.run.smithi088.stdout: 262177 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Oct 18 16:16 /home/ubuntu/cephtest/archive 2023-10-18T16:17:24.285 INFO:teuthology.orchestra.run.smithi088.stdout: 262179 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 18 16:16 /home/ubuntu/cephtest/archive/syslog 2023-10-18T16:17:24.286 INFO:teuthology.orchestra.run.smithi088.stdout: 262153 112 -rw-rw-r-- 1 ubuntu ubuntu 113278 Oct 18 16:16 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-18T16:17:24.286 INFO:teuthology.orchestra.run.smithi088.stdout: 262146 4 -rw-rw-rw- 1 ubuntu ubuntu 799 Oct 18 16:16 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2023-10-18T16:17:24.286 INFO:teuthology.orchestra.run.smithi088.stdout: 262145 116 -rw-rw-rw- 1 ubuntu ubuntu 116613 Oct 18 16:16 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2023-10-18T16:17:24.287 INFO:teuthology.orchestra.run.smithi088.stdout: 262178 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 18 16:09 /home/ubuntu/cephtest/archive/coredump 2023-10-18T16:17:24.287 INFO:teuthology.orchestra.run.smithi088.stdout: 262152 17648 -rw------- 1 root root 300449792 Oct 18 16:09 /home/ubuntu/cephtest/archive/coredump/1697645373.94618.core 2023-10-18T16:17:24.287 INFO:teuthology.orchestra.run.smithi088.stdout: 262148 14432 -rw------- 1 root root 299405312 Oct 18 16:09 /home/ubuntu/cephtest/archive/coredump/1697645358.94619.core 2023-10-18T16:17:24.287 INFO:teuthology.orchestra.run.smithi088.stdout: 262147 14724 -rw------- 1 root root 299401216 Oct 18 16:08 /home/ubuntu/cephtest/archive/coredump/1697645328.94620.core 2023-10-18T16:17:24.288 INFO:teuthology.orchestra.run.smithi088.stdout: 393224 4 drwxrwxr-x 2 ubuntu ubuntu 4096 Oct 18 16:16 /home/ubuntu/cephtest/archive/audit 2023-10-18T16:17:24.288 INFO:teuthology.orchestra.run.smithi088.stdout: 393310 248 -rw------- 1 ubuntu ubuntu 253256 Oct 18 16:16 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2023-10-18T16:17:24.289 INFO:teuthology.orchestra.run.smithi151.stdout: 262149 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Oct 18 16:16 /home/ubuntu/cephtest 2023-10-18T16:17:24.289 INFO:teuthology.orchestra.run.smithi151.stdout: 262150 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Oct 18 16:16 /home/ubuntu/cephtest/archive 2023-10-18T16:17:24.290 INFO:teuthology.orchestra.run.smithi151.stdout: 262152 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 18 16:16 /home/ubuntu/cephtest/archive/syslog 2023-10-18T16:17:24.290 INFO:teuthology.orchestra.run.smithi151.stdout: 262157 104 -rw-rw-r-- 1 ubuntu ubuntu 105635 Oct 18 16:16 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-18T16:17:24.290 INFO:teuthology.orchestra.run.smithi151.stdout: 262146 4 -rw-rw-rw- 1 ubuntu ubuntu 626 Oct 18 16:16 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2023-10-18T16:17:24.291 INFO:teuthology.orchestra.run.smithi151.stdout: 262145 104 -rw-rw-rw- 1 ubuntu ubuntu 106278 Oct 18 16:16 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2023-10-18T16:17:24.291 INFO:teuthology.orchestra.run.smithi151.stdout: 262151 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 18 16:12 /home/ubuntu/cephtest/archive/coredump 2023-10-18T16:17:24.291 INFO:teuthology.orchestra.run.smithi151.stdout: 262148 16556 -rw------- 1 root root 299401216 Oct 18 16:08 /home/ubuntu/cephtest/archive/coredump/1697645313.94971.core 2023-10-18T16:17:24.291 INFO:teuthology.orchestra.run.smithi151.stdout: 262147 13600 -rw------- 1 root root 299405312 Oct 18 16:07 /home/ubuntu/cephtest/archive/coredump/1697645261.94970.core 2023-10-18T16:17:24.292 INFO:teuthology.orchestra.run.smithi151.stdout: 262162 48312 -rw------- 1 root root 293380096 Oct 18 16:12 /home/ubuntu/cephtest/archive/coredump/1697645573.90915.core 2023-10-18T16:17:24.292 INFO:teuthology.orchestra.run.smithi151.stdout: 262161 14784 -rw------- 1 root root 299405312 Oct 18 16:09 /home/ubuntu/cephtest/archive/coredump/1697645343.94969.core 2023-10-18T16:17:24.292 INFO:teuthology.orchestra.run.smithi151.stdout: 393222 4 drwxrwxr-x 2 ubuntu ubuntu 4096 Oct 18 16:16 /home/ubuntu/cephtest/archive/audit 2023-10-18T16:17:24.292 INFO:teuthology.orchestra.run.smithi151.stdout: 393308 236 -rw------- 1 ubuntu ubuntu 241006 Oct 18 16:16 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2023-10-18T16:17:24.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T16:17:24.294 INFO:teuthology.orchestra.run.smithi088.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-10-18T16:17:24.295 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/task/internal/__init__.py", line 420, in archive fetch_binaries_for_coredumps(path, rem) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/task/internal/__init__.py", line 345, in fetch_binaries_for_coredumps remote_path = remote.sh(['which', dump_program]).rstrip() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/remote.py", line 96, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi088 with status 1: "which 'ceph-mds -f --cluster ceph -i a'" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi088 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-10-18T16:17:24.302 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-18T16:17:24.314 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-18T16:17:24.338 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi088 with status 1: "which 'ceph-mds -f --cluster ceph -i a'" 2023-10-18T16:17:24.339 INFO:teuthology.nuke:Checking targets against current locks 2023-10-18T16:17:24.341 INFO:teuthology.orchestra.run.smithi151.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-10-18T16:17:24.369 DEBUG:teuthology.nuke:shortname: smithi088 2023-10-18T16:17:24.370 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-18T16:17:24.394 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi088.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2023-10-16_12:48:28-fs-wip-vshankar-testing-pacific-20231016.091648-testing-default-smithi/7429067', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-18 15:48:36.620568', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDM0J7RzHKtMvi+MMC04dICQhoKb8N07LvmP9S2NyG0Ljq9D1SSqj3y5PRJLxhMdmsMjbaiAzPfrTsQdrK9x9Ho='} 2023-10-18T16:17:24.397 DEBUG:teuthology.nuke:shortname: smithi151 2023-10-18T16:17:24.398 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-18T16:17:24.423 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi151.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2023-10-16_12:48:28-fs-wip-vshankar-testing-pacific-20231016.091648-testing-default-smithi/7429067', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-18 15:48:36.618906', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBbNF+1REZY6QhkJY1gSEnsFbRSKC+civjX7dXcQZ0VlbSAYkufd5XtNVsJX3hiJyfcxEIIh87QC/wsq4npNqzM='} 2023-10-18T16:17:24.449 INFO:teuthology.orchestra.console.smithi088:Power off 2023-10-18T16:17:24.450 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-18T16:17:24.473 DEBUG:teuthology.orchestra.console.smithi088:power off output: Chassis Power Control: Down/Off 2023-10-18T16:17:24.474 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T16:17:24.487 INFO:teuthology.orchestra.console.smithi151:Power off 2023-10-18T16:17:24.490 DEBUG:teuthology.orchestra.console.smithi151:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-18T16:17:24.500 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is on 2023-10-18T16:17:24.515 DEBUG:teuthology.orchestra.console.smithi151:power off output: Chassis Power Control: Down/Off 2023-10-18T16:17:24.515 DEBUG:teuthology.orchestra.console.smithi151:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T16:17:24.537 DEBUG:teuthology.orchestra.console.smithi151:check power output: Chassis Power is on 2023-10-18T16:17:28.502 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T16:17:28.539 DEBUG:teuthology.orchestra.console.smithi151:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T16:17:28.613 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is on 2023-10-18T16:17:28.649 DEBUG:teuthology.orchestra.console.smithi151:check power output: Chassis Power is on 2023-10-18T16:17:32.619 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T16:17:32.650 DEBUG:teuthology.orchestra.console.smithi151:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T16:17:32.729 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is on 2023-10-18T16:17:32.760 DEBUG:teuthology.orchestra.console.smithi151:check power output: Chassis Power is off 2023-10-18T16:17:32.861 INFO:teuthology.orchestra.console.smithi151:Power off completed 2023-10-18T16:17:36.731 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T16:17:36.842 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is off 2023-10-18T16:17:36.943 INFO:teuthology.orchestra.console.smithi088:Power off completed 2023-10-18T16:17:37.049 INFO:teuthology.run:Summary data: description: fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} duration: 1232.486735343933 failure_reason: 'Command failed on smithi088 with status 124: ''sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json''' flavor: default owner: scheduled_vshankar@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=7a6392e832b64f928b10d2794f9bf3f7 status: fail success: false 2023-10-18T16:17:37.049 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-18T16:17:37.130 INFO:teuthology.run:FAIL